Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:58:27.344463 398 test_util.cc:268] Using random seed: 364089404
I20240503 07:58:27.351308 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:27.351413 398 ts_itest-base.cc:113] --------------
I20240503 07:58:27.351452 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:58:27.351480 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:27.351496 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:27Z Disabled control of system clock
I20240503 07:58:27.360215 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37543
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37543
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:58:27.464149 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:27.464385 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:27.464442 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:27.467696 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:27.467782 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:27.467813 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:27.467841 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:27.467868 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:27.471704 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37543
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:27.472606 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:27.473781 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:27.481932 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.481917 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.482511 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:27.483129 410 server_base.cc:1034] running on GCE node
I20240503 07:58:27.483628 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:27.484586 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:27.485739 410 hybrid_clock.cc:648] HybridClock initialized: now 1714723107485698 us; error 52 us; skew 500 ppm
I20240503 07:58:27.487706 410 webserver.cc:460] Webserver started at http://127.0.99.190:46577/ using document root <none> and password file <none>
I20240503 07:58:27.488262 410 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:27.488327 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:27.488502 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:27.490188 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "10aa3f4ccc5645d4874fe1b46731a58f"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.490686 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "10aa3f4ccc5645d4874fe1b46731a58f"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.494153 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:27.496609 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:27.497649 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:27.497776 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "10aa3f4ccc5645d4874fe1b46731a58f"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.497864 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:27.519441 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:27.520141 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:27.520316 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:27.526948 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37543
I20240503 07:58:27.526975 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37543 every 8 connection(s)
I20240503 07:58:27.527928 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:27.530647 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:27.535598 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 410
I20240503 07:58:27.535650 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f: Bootstrap starting.
I20240503 07:58:27.535730 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:27.537783 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:27.538513 477 log.cc:824] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:27.540424 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f: No bootstrap required, opened a new log
I20240503 07:58:27.543592 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } }
I20240503 07:58:27.543773 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:27.543843 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aa3f4ccc5645d4874fe1b46731a58f, State: Initialized, Role: FOLLOWER
I20240503 07:58:27.544282 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } }
I20240503 07:58:27.544407 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:27.544469 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:27.544564 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:27.600230 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } }
I20240503 07:58:27.600769 477 leader_election.cc:302] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10aa3f4ccc5645d4874fe1b46731a58f; no voters:
I20240503 07:58:27.601114 477 leader_election.cc:288] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:27.601230 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:27.601447 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [term 1 LEADER]: Becoming Leader. State: Replica: 10aa3f4ccc5645d4874fe1b46731a58f, State: Running, Role: LEADER
I20240503 07:58:27.601848 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } }
I20240503 07:58:27.602391 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:27.603679 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } } }
I20240503 07:58:27.603802 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:27.603660 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10aa3f4ccc5645d4874fe1b46731a58f. Latest consensus state: current_term: 1 leader_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10aa3f4ccc5645d4874fe1b46731a58f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37543 } } }
I20240503 07:58:27.604473 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:27.604254 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:27.607784 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:27.613057 491 catalog_manager.cc:1347] Generated new cluster ID: 28a0b7bc8e394707b517af6b63a88431
I20240503 07:58:27.613140 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:27.630280 491 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:27.631630 491 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:27.643289 491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 10aa3f4ccc5645d4874fe1b46731a58f: Generated new TSK 0
I20240503 07:58:27.643991 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:27.658875 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:58:27.768819 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:27.769047 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:27.769102 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:27.772248 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:27.772320 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:27.772406 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:27.776018 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:27.776885 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:27.777983 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:27.786602 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.787151 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.787299 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:27.787932 501 server_base.cc:1034] running on GCE node
I20240503 07:58:27.788388 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:27.788990 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:27.790169 501 hybrid_clock.cc:648] HybridClock initialized: now 1714723107790126 us; error 64 us; skew 500 ppm
I20240503 07:58:27.792325 501 webserver.cc:460] Webserver started at http://127.0.99.129:43113/ using document root <none> and password file <none>
I20240503 07:58:27.792857 501 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:27.792912 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:27.793110 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:27.794770 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "27d51a644d914549b3dfb3f104d396e8"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.795315 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "27d51a644d914549b3dfb3f104d396e8"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.798774 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:27.801147 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:27.802229 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:27.802353 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "27d51a644d914549b3dfb3f104d396e8"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.802449 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:27.820251 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:27.820905 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:27.821071 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:27.821646 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:27.822546 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:27.822592 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:27.822644 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:27.822671 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:27.830626 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36481
I20240503 07:58:27.830650 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36481 every 8 connection(s)
I20240503 07:58:27.831571 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:27.834277 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 501
I20240503 07:58:27.834388 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:27.836573 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:27.841488 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:27.841751 629 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:27.842633 629 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:27.844331 441 ts_manager.cc:194] Registered new tserver with Master: 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:27.845536 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47539
W20240503 07:58:27.946287 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:27.946513 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:27.946573 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:27.949764 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:27.949837 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:27.949930 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:27.953505 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:27.954396 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:27.955490 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:27.963824 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.964395 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:27.964994 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:27.965554 632 server_base.cc:1034] running on GCE node
I20240503 07:58:27.965953 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:27.966495 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:27.967660 632 hybrid_clock.cc:648] HybridClock initialized: now 1714723107967632 us; error 52 us; skew 500 ppm
I20240503 07:58:27.969635 632 webserver.cc:460] Webserver started at http://127.0.99.130:41381/ using document root <none> and password file <none>
I20240503 07:58:27.970163 632 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:27.970220 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:27.970388 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:27.972116 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7a36d94f14d04d74b632bc75d1779bc3"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.972574 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7a36d94f14d04d74b632bc75d1779bc3"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.975931 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:27.978137 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:27.979173 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:27.979285 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7a36d94f14d04d74b632bc75d1779bc3"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:58:27.979367 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.003626 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.004279 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.004446 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.005038 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.005997 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.006053 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.006107 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.006129 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.015551 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38831
I20240503 07:58:28.015605 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38831 every 8 connection(s)
I20240503 07:58:28.016505 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:28.022877 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 632
I20240503 07:58:28.023015 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:28.025182 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:28.025601 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.025822 760 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.026402 760 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.027390 441 ts_manager.cc:194] Registered new tserver with Master: 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:28.027941 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:60471
W20240503 07:58:28.132184 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:28.132407 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:28.132472 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:28.135566 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:28.135628 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:28.135716 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:28.139354 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:28.140236 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:28.141345 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:28.149796 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.150018 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.150614 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:28.151222 763 server_base.cc:1034] running on GCE node
I20240503 07:58:28.151572 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:28.152094 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:28.153245 763 hybrid_clock.cc:648] HybridClock initialized: now 1714723108153232 us; error 28 us; skew 500 ppm
I20240503 07:58:28.155736 763 webserver.cc:460] Webserver started at http://127.0.99.131:34587/ using document root <none> and password file <none>
I20240503 07:58:28.156284 763 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:28.156343 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:28.156519 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:28.158154 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5cef4772b54342fe9d418f103c30fc33"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.158640 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5cef4772b54342fe9d418f103c30fc33"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.161989 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:28.164075 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.165041 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:28.165158 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5cef4772b54342fe9d418f103c30fc33"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.165246 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.185616 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.186228 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.186376 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.186902 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.187901 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.187954 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.188009 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.188036 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.197460 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44423
I20240503 07:58:28.197485 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44423 every 8 connection(s)
I20240503 07:58:28.198369 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:28.201133 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 763
I20240503 07:58:28.201218 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:28.203368 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:28.207790 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.208025 891 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.208611 891 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.209611 441 ts_manager.cc:194] Registered new tserver with Master: 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:28.210160 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47835
W20240503 07:58:28.318295 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:28.318522 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:28.318581 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:28.321610 894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:28.321671 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:28.321754 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:28.325353 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:28.326232 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:28.327370 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:28.335492 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.336114 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.336792 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:28.337579 894 server_base.cc:1034] running on GCE node
I20240503 07:58:28.337970 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:28.338577 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:28.339741 894 hybrid_clock.cc:648] HybridClock initialized: now 1714723108339711 us; error 42 us; skew 500 ppm
I20240503 07:58:28.341895 894 webserver.cc:460] Webserver started at http://127.0.99.132:45871/ using document root <none> and password file <none>
I20240503 07:58:28.342456 894 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:28.342514 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:28.342679 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:28.344435 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.344903 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.348455 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 07:58:28.350710 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.351766 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:28.351868 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.351951 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.379182 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.379818 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.379977 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.380529 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.381430 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.381479 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.381531 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.381556 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.391113 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39363
I20240503 07:58:28.391131 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39363 every 8 connection(s)
I20240503 07:58:28.392064 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:28.401078 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.401335 1022 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.401329 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 894
I20240503 07:58:28.401444 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:28.401949 1022 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.402972 441 ts_manager.cc:194] Registered new tserver with Master: 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:28.403647 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:59983
I20240503 07:58:28.403700 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:58:28.511420 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:28.511651 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:28.511701 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:28.514781 1025 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:28.514858 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:28.514971 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:28.518754 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:28.519798 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:28.520916 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:28.530472 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.531050 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.531219 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:28.531898 1025 server_base.cc:1034] running on GCE node
I20240503 07:58:28.532364 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:28.532999 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:28.534276 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714723108534262 us; error 36 us; skew 500 ppm
I20240503 07:58:28.536412 1025 webserver.cc:460] Webserver started at http://127.0.99.133:42073/ using document root <none> and password file <none>
I20240503 07:58:28.536957 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:28.537014 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:28.537215 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:28.539265 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9e85f3485213464fb71cb81d65aedbad"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.539790 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9e85f3485213464fb71cb81d65aedbad"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.543769 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:28.546160 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.547250 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 07:58:28.547354 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9e85f3485213464fb71cb81d65aedbad"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.547441 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.580019 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.580633 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.580781 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.581295 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.582193 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.582240 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.582293 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.582316 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.591292 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33799
I20240503 07:58:28.591315 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33799 every 8 connection(s)
I20240503 07:58:28.592216 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:28.599786 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 1025
I20240503 07:58:28.599876 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:28.600389 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.600605 1153 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.601207 1153 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.601958 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:28.602198 441 ts_manager.cc:194] Registered new tserver with Master: 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799)
I20240503 07:58:28.602792 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:36441
W20240503 07:58:28.717159 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:28.717386 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:28.717447 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:28.720520 1156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:28.720583 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:28.720669 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:28.724260 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:28.725116 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:28.726181 1156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:28.734587 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.735078 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.735725 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:28.736234 1156 server_base.cc:1034] running on GCE node
I20240503 07:58:28.736582 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:28.737104 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:28.738255 1156 hybrid_clock.cc:648] HybridClock initialized: now 1714723108738237 us; error 34 us; skew 500 ppm
I20240503 07:58:28.740068 1156 webserver.cc:460] Webserver started at http://127.0.99.134:44103/ using document root <none> and password file <none>
I20240503 07:58:28.740589 1156 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:28.740640 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:28.740813 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:28.742830 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.743402 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.747363 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 07:58:28.749518 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.750500 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:28.750625 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.750708 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.772085 1156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.772701 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.772852 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.773386 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.774247 1156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.774293 1156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.774344 1156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.774369 1156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.784097 1156 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34213
I20240503 07:58:28.784121 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34213 every 8 connection(s)
I20240503 07:58:28.785050 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:28.788133 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 1156
I20240503 07:58:28.788246 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:58:28.790413 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:28.794588 1284 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.794821 1284 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.795397 1284 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.796523 441 ts_manager.cc:194] Registered new tserver with Master: 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213)
I20240503 07:58:28.797263 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:44417
I20240503 07:58:28.848217 629 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
W20240503 07:58:28.899399 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:28.899626 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:28.899672 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:28.902698 1287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:28.902767 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:28.902853 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:28.906749 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:28.907717 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:28.908843 1287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:28.917668 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.918347 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:28.919020 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:28.919641 1287 server_base.cc:1034] running on GCE node
I20240503 07:58:28.920073 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:28.920693 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:28.921847 1287 hybrid_clock.cc:648] HybridClock initialized: now 1714723108921830 us; error 39 us; skew 500 ppm
I20240503 07:58:28.923843 1287 webserver.cc:460] Webserver started at http://127.0.99.135:41273/ using document root <none> and password file <none>
I20240503 07:58:28.924412 1287 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:28.924472 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:28.924643 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:28.927331 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b8f135eb22964d4a8d8300956bce8017"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.927809 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b8f135eb22964d4a8d8300956bce8017"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.931241 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:28.933459 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.934466 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:28.934576 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b8f135eb22964d4a8d8300956bce8017"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:28.934665 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:28.949359 1287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:28.950023 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:28.950162 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:28.950686 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:28.951683 1287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:28.951730 1287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.951783 1287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:28.951807 1287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:28.962452 1287 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35843
I20240503 07:58:28.962471 1414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35843 every 8 connection(s)
I20240503 07:58:28.963483 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:28.966358 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 1287
I20240503 07:58:28.966598 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:58:28.970533 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37543
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:28.973681 1415 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:28.973912 1415 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:28.974493 1415 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:28.975471 441 ts_manager.cc:194] Registered new tserver with Master: b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843)
I20240503 07:58:28.976006 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:42015
I20240503 07:58:29.030751 760 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
W20240503 07:58:29.081173 1418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:29.081418 1418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:29.081467 1418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:29.084636 1418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:29.084709 1418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:29.084800 1418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:29.088498 1418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.1418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:29.089435 1418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:29.090646 1418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:29.099864 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:29.100338 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:29.100932 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:29.101600 1418 server_base.cc:1034] running on GCE node
I20240503 07:58:29.102097 1418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:29.102797 1418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:29.103950 1418 hybrid_clock.cc:648] HybridClock initialized: now 1714723109103933 us; error 55 us; skew 500 ppm
I20240503 07:58:29.105898 1418 webserver.cc:460] Webserver started at http://127.0.99.136:41891/ using document root <none> and password file <none>
I20240503 07:58:29.106469 1418 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:29.106525 1418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:29.106693 1418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:29.108348 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "164b774d3a3346bca3572f022142e17c"
format_stamp: "Formatted at 2024-05-03 07:58:29 on dist-test-slave-11df"
I20240503 07:58:29.108789 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "164b774d3a3346bca3572f022142e17c"
format_stamp: "Formatted at 2024-05-03 07:58:29 on dist-test-slave-11df"
I20240503 07:58:29.112097 1418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 07:58:29.114447 1432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:29.115605 1418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:29.115729 1418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "164b774d3a3346bca3572f022142e17c"
format_stamp: "Formatted at 2024-05-03 07:58:29 on dist-test-slave-11df"
I20240503 07:58:29.115818 1418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:29.140949 1418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:29.141613 1418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:29.141763 1418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:29.142283 1418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:29.143256 1418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:29.143321 1418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:29.143383 1418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:29.143407 1418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:29.153481 1418 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:45427
I20240503 07:58:29.153497 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:45427 every 8 connection(s)
I20240503 07:58:29.154444 1418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:29.156412 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 1418
I20240503 07:58:29.156533 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:58:29.163178 1546 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:29.163393 1546 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:29.163959 1546 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:29.164903 441 ts_manager.cc:194] Registered new tserver with Master: 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427)
I20240503 07:58:29.165439 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:36769
I20240503 07:58:29.171013 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:29.181620 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:29.181703 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 1025
I20240503 07:58:29.187248 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 1156
I20240503 07:58:29.194326 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 1287
I20240503 07:58:29.200716 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 1418
I20240503 07:58:29.212085 891 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:58:29.407171 1022 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:58:32.958019 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41884:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:32.980935 563 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.980934 694 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.980935 812 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2bc8609e4945148a50dde608502f67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.981912 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.981912 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.981926 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.980935 825 tablet_service.cc:1467] Processing CreateTablet for tablet 97138ab5e3394c41b83382a22df0a198 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.983201 693 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2bc8609e4945148a50dde608502f67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.983331 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97138ab5e3394c41b83382a22df0a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.983685 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.984099 956 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.984923 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.987248 955 tablet_service.cc:1467] Processing CreateTablet for tablet 97138ab5e3394c41b83382a22df0a198 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.987248 561 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2bc8609e4945148a50dde608502f67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.987488 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97138ab5e3394c41b83382a22df0a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.990612 562 tablet_service.cc:1467] Processing CreateTablet for tablet 97138ab5e3394c41b83382a22df0a198 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:32.990959 1566 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:32.991462 1567 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:32.992745 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.993120 1566 tablet_bootstrap.cc:654] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:32.993440 1567 tablet_bootstrap.cc:654] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:32.993954 1566 log.cc:824] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:32.994102 1567 log.cc:824] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:32.994305 1568 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:32.995748 1567 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:32.995748 1566 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:32.995918 1567 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:58:32.995918 1566 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:32.996639 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97138ab5e3394c41b83382a22df0a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.999266 1571 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:32.999625 1567 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:32.999627 1566 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:32.999833 1566 raft_consensus.cc:374] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:32.999833 1567 raft_consensus.cc:374] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:32.999881 1566 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:32.999881 1567 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.000347 1566 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.000347 1567 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.001205 1566 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:58:33.001205 1567 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20240503 07:58:33.001355 1571 tablet_bootstrap.cc:654] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.002202 1571 log.cc:824] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:33.003393 1568 tablet_bootstrap.cc:654] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.003744 1567 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:33.003890 1571 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:33.004061 1571 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:33.004096 1566 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:33.004428 1568 log.cc:824] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:33.005013 1566 tablet_bootstrap.cc:654] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.006201 1566 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:33.006251 1568 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:33.006278 1566 ts_tablet_manager.cc:1367] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:33.006424 1568 ts_tablet_manager.cc:1367] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240503 07:58:33.006855 1566 raft_consensus.cc:348] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.006989 1566 raft_consensus.cc:374] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.007025 1566 raft_consensus.cc:729] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.007159 1566 consensus_queue.cc:260] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.007428 1566 ts_tablet_manager.cc:1398] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:33.007403 1571 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.007597 1571 raft_consensus.cc:374] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.007640 1571 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.007874 1567 tablet_bootstrap.cc:654] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.008297 1571 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.009006 1567 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:33.009078 1567 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:58:33.009122 1571 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:33.009443 1571 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:33.009558 1567 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.009649 1567 raft_consensus.cc:374] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.009676 1567 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.009814 1567 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.009958 1568 raft_consensus.cc:348] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.010133 1568 raft_consensus.cc:374] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.010159 1567 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:58:33.010179 1568 raft_consensus.cc:729] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.010277 1571 tablet_bootstrap.cc:654] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.011272 1571 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:33.011348 1571 ts_tablet_manager.cc:1367] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:33.011826 1571 raft_consensus.cc:348] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.011926 1571 raft_consensus.cc:374] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.011955 1571 raft_consensus.cc:729] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.012070 1571 consensus_queue.cc:260] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.012385 1571 ts_tablet_manager.cc:1398] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.012545 1571 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:33.013376 1571 tablet_bootstrap.cc:654] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.013473 1568 consensus_queue.cc:260] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.014386 1568 ts_tablet_manager.cc:1398] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20240503 07:58:33.014485 1571 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:33.014559 1571 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:33.014669 1568 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:33.014978 1571 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.015085 1571 raft_consensus.cc:374] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.015125 1571 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.015226 1571 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.015476 1571 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.016208 1568 tablet_bootstrap.cc:654] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.017223 1568 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:33.017295 1568 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:58:33.017762 1568 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.017864 1568 raft_consensus.cc:374] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.017889 1568 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.017998 1568 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.018222 1568 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:33.019393 1578 raft_consensus.cc:482] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.019552 1578 raft_consensus.cc:504] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.020732 1578 leader_election.cc:288] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
W20240503 07:58:33.020922 761 tablet.cc:2369] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:33.024979 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:33.025200 976 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 0.
I20240503 07:58:33.025301 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:33.025543 845 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 0.
I20240503 07:58:33.025612 519 leader_election.cc:302] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.025805 1578 raft_consensus.cc:2781] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.025883 1578 raft_consensus.cc:482] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.025933 1578 raft_consensus.cc:3037] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.026716 1578 raft_consensus.cc:504] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.027047 1578 leader_election.cc:288] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Requested vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:33.027297 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:33.027326 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:33.027408 976 raft_consensus.cc:3037] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.027424 845 raft_consensus.cc:3037] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.028160 845 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:33.028172 976 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:33.028412 518 leader_election.cc:302] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 5cef4772b54342fe9d418f103c30fc33; no voters:
I20240503 07:58:33.028568 1578 raft_consensus.cc:2781] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.028640 1578 raft_consensus.cc:686] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 1 LEADER]: Becoming Leader. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Running, Role: LEADER
I20240503 07:58:33.029016 1578 consensus_queue.cc:237] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.031675 435 catalog_manager.cc:5484] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "27d51a644d914549b3dfb3f104d396e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:33.044881 1578 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.044984 1578 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.045372 1578 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.046137 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:33.046279 845 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 0.
I20240503 07:58:33.046537 518 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 5cef4772b54342fe9d418f103c30fc33; no voters:
I20240503 07:58:33.046696 1578 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.046782 1578 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.046809 1578 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.047806 1578 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.048157 1578 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.048437 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:33.048554 845 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.049337 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:33.049413 845 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:33.049347 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:33.049564 713 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.049760 518 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 5cef4772b54342fe9d418f103c30fc33; no voters:
I20240503 07:58:33.049932 1578 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.050016 1578 raft_consensus.cc:686] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 1 LEADER]: Becoming Leader. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Running, Role: LEADER
I20240503 07:58:33.050151 1578 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:33.050568 713 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:33.051821 435 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "27d51a644d914549b3dfb3f104d396e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:33.086359 630 tablet.cc:2369] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:33.111462 1579 raft_consensus.cc:482] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.111600 1579 raft_consensus.cc:504] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.112641 1579 leader_election.cc:288] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:33.117246 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca5e6693b24d0e8f3d9c37a62a1bca" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:33.117393 976 raft_consensus.cc:2454] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:33.117347 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca5e6693b24d0e8f3d9c37a62a1bca" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:33.117545 583 raft_consensus.cc:2454] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:33.117775 650 leader_election.cc:302] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.118006 1579 raft_consensus.cc:2781] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.118088 1579 raft_consensus.cc:482] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.118147 1579 raft_consensus.cc:3037] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.119021 1579 raft_consensus.cc:504] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.119382 1579 leader_election.cc:288] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:33.119758 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca5e6693b24d0e8f3d9c37a62a1bca" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:33.119758 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca5e6693b24d0e8f3d9c37a62a1bca" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:33.119882 976 raft_consensus.cc:3037] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.119882 583 raft_consensus.cc:3037] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.120786 583 raft_consensus.cc:2454] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:33.120793 976 raft_consensus.cc:2454] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:33.121076 650 leader_election.cc:302] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.121260 1579 raft_consensus.cc:2781] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.121374 1579 raft_consensus.cc:686] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:33.121678 1579 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:33.124523 435 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:33.146747 1023 tablet.cc:2369] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:33.203047 892 tablet.cc:2369] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:33.239799 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:33.240026 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:33.243041 1592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:58:33.332221 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.332229 976 raft_consensus.cc:1261] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.332890 1581 consensus_queue.cc:1035] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.334638 975 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.335259 1573 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.337926 845 raft_consensus.cc:1261] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.338537 583 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.339257 713 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.339646 1578 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.339669 1573 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.339993 1578 consensus_queue.cc:1035] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.343662 1581 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.345024 1625 mvcc.cc:205] Tried to move back new op lower bound from 7023505872212295680 to 7023505871348760576. Current Snapshot: MvccSnapshot[applied={T|T < 7023505872212295680}]
I20240503 07:58:33.349217 1623 mvcc.cc:205] Tried to move back new op lower bound from 7023505872232296448 to 7023505871348760576. Current Snapshot: MvccSnapshot[applied={T|T < 7023505872212295680}]
I20240503 07:58:33.352278 1622 mvcc.cc:205] Tried to move back new op lower bound from 7023505872201048064 to 7023505870969163776. Current Snapshot: MvccSnapshot[applied={T|T < 7023505872201048064}]
I20240503 07:58:33.380362 1635 mvcc.cc:205] Tried to move back new op lower bound from 7023505872225959936 to 7023505871054864384. Current Snapshot: MvccSnapshot[applied={T|T < 7023505872200765440}]
INSERT report
rows total: 2000
time total: 111.933 ms
time per row: 0.0559666 ms
I20240503 07:58:33.405165 433 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41884:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:33.410326 694 tablet_service.cc:1467] Processing CreateTablet for tablet 33a8cbbafd86402a9a3bf9e730b60c0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.410326 812 tablet_service.cc:1467] Processing CreateTablet for tablet 33a8cbbafd86402a9a3bf9e730b60c0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.410514 956 tablet_service.cc:1467] Processing CreateTablet for tablet 33a8cbbafd86402a9a3bf9e730b60c0e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.410624 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a8cbbafd86402a9a3bf9e730b60c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.410624 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a8cbbafd86402a9a3bf9e730b60c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.410792 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a8cbbafd86402a9a3bf9e730b60c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.412381 1568 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:33.412679 1567 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:33.413187 1568 tablet_bootstrap.cc:654] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.413178 694 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f5e5ffc9946639c4da0129d6d479e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.413374 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f5e5ffc9946639c4da0129d6d479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.413455 1567 tablet_bootstrap.cc:654] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.413584 561 tablet_service.cc:1467] Processing CreateTablet for tablet f42fb6ac38fc4b4782814cae0891b0cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.413808 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42fb6ac38fc4b4782814cae0891b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.414016 1566 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:33.414441 1567 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:33.414512 1567 ts_tablet_manager.cc:1367] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:58:33.414794 1566 tablet_bootstrap.cc:654] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.414966 1567 raft_consensus.cc:348] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.415061 1567 raft_consensus.cc:374] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.415092 1567 raft_consensus.cc:729] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.415194 1567 consensus_queue.cc:260] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.415361 694 tablet_service.cc:1467] Processing CreateTablet for tablet f42fb6ac38fc4b4782814cae0891b0cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.415509 1567 ts_tablet_manager.cc:1398] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.415562 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42fb6ac38fc4b4782814cae0891b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.415786 825 tablet_service.cc:1467] Processing CreateTablet for tablet f42fb6ac38fc4b4782814cae0891b0cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.415922 1571 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:33.415997 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f42fb6ac38fc4b4782814cae0891b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.416754 1571 tablet_bootstrap.cc:654] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.417549 956 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f5e5ffc9946639c4da0129d6d479e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.415686 1567 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:33.417733 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f5e5ffc9946639c4da0129d6d479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.418449 1567 tablet_bootstrap.cc:654] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.419042 812 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f5e5ffc9946639c4da0129d6d479e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.419248 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f5e5ffc9946639c4da0129d6d479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.420293 1571 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:33.420356 1571 ts_tablet_manager.cc:1367] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:58:33.420732 1571 raft_consensus.cc:348] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.420848 1571 raft_consensus.cc:374] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.420882 1571 raft_consensus.cc:729] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.420998 1571 consensus_queue.cc:260] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.421224 1571 ts_tablet_manager.cc:1398] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.421334 1566 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:33.421418 1566 ts_tablet_manager.cc:1367] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 07:58:33.421849 1566 raft_consensus.cc:348] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.421918 1568 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:33.421949 1566 raft_consensus.cc:374] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.421977 1566 raft_consensus.cc:729] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.421989 1568 ts_tablet_manager.cc:1367] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:58:33.422106 1566 consensus_queue.cc:260] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.422354 1566 ts_tablet_manager.cc:1398] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:33.422398 1568 raft_consensus.cc:348] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.422482 1568 raft_consensus.cc:374] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.422510 1568 raft_consensus.cc:729] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.422533 1566 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:33.422618 1568 consensus_queue.cc:260] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.422881 1568 ts_tablet_manager.cc:1398] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.423086 1568 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:33.423517 1566 tablet_bootstrap.cc:654] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.423892 1568 tablet_bootstrap.cc:654] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.424639 1566 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:33.424721 1566 ts_tablet_manager.cc:1367] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:33.425134 1566 raft_consensus.cc:348] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.425220 1566 raft_consensus.cc:374] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.425249 1566 raft_consensus.cc:729] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.425371 1567 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:33.425387 1566 consensus_queue.cc:260] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.425491 1567 ts_tablet_manager.cc:1367] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:58:33.425732 1566 ts_tablet_manager.cc:1398] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:33.425876 1566 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:33.426007 1567 raft_consensus.cc:348] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.426118 1567 raft_consensus.cc:374] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.426151 1567 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.426321 1567 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.426568 1567 ts_tablet_manager.cc:1398] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.426714 1567 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:33.427386 1566 tablet_bootstrap.cc:654] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.427762 1567 tablet_bootstrap.cc:654] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.428354 1568 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:33.428467 1568 ts_tablet_manager.cc:1367] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:58:33.428831 1567 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:33.428891 1566 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:33.428903 1567 ts_tablet_manager.cc:1367] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:33.428963 1566 ts_tablet_manager.cc:1367] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:58:33.429311 1567 raft_consensus.cc:348] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.429342 1566 raft_consensus.cc:348] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.429409 1567 raft_consensus.cc:374] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.429428 1566 raft_consensus.cc:374] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.429443 1567 raft_consensus.cc:729] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.429396 1568 raft_consensus.cc:348] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.429456 1566 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.429479 1568 raft_consensus.cc:374] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.429508 1568 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.429543 1567 consensus_queue.cc:260] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.429589 1566 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.429605 1568 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.429786 1567 ts_tablet_manager.cc:1398] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.429829 1568 ts_tablet_manager.cc:1398] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:33.429843 1566 ts_tablet_manager.cc:1398] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:33.636884 1573 raft_consensus.cc:482] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.637034 1573 raft_consensus.cc:504] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.637518 1573 leader_election.cc:288] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:33.638502 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42fb6ac38fc4b4782814cae0891b0cc" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:33.638696 583 raft_consensus.cc:2454] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:33.639075 649 leader_election.cc:302] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:33.639282 1573 raft_consensus.cc:2781] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.639348 1573 raft_consensus.cc:482] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.639382 1573 raft_consensus.cc:3037] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.640394 1573 raft_consensus.cc:504] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.640715 1573 leader_election.cc:288] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:33.641112 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42fb6ac38fc4b4782814cae0891b0cc" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:33.641248 583 raft_consensus.cc:3037] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.641824 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42fb6ac38fc4b4782814cae0891b0cc" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:33.641863 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f42fb6ac38fc4b4782814cae0891b0cc" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:33.641961 844 raft_consensus.cc:2454] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:33.642199 583 raft_consensus.cc:2454] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:33.642412 649 leader_election.cc:302] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3; no voters: 5cef4772b54342fe9d418f103c30fc33
I20240503 07:58:33.642560 1573 raft_consensus.cc:2781] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.642642 1573 raft_consensus.cc:686] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:33.642791 1573 consensus_queue.cc:237] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:33.644492 436 catalog_manager.cc:5484] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:33.718322 1582 raft_consensus.cc:482] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.718452 1582 raft_consensus.cc:504] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.719511 1582 leader_election.cc:288] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.723124 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:33.723286 713 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:33.723302 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:33.723446 845 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:33.723659 911 leader_election.cc:302] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.723917 1582 raft_consensus.cc:2781] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.723974 1582 raft_consensus.cc:482] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.724000 1582 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.724781 1582 raft_consensus.cc:504] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.725104 1582 leader_election.cc:288] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.725431 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:33.725471 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:33.725545 713 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.725603 845 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.726302 845 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:33.726301 713 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:33.726590 911 leader_election.cc:302] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.726747 1582 raft_consensus.cc:2781] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.726907 1582 raft_consensus.cc:686] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:33.727258 1582 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.729522 436 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:33.743042 1582 raft_consensus.cc:482] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.743178 1582 raft_consensus.cc:504] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.743558 1582 leader_election.cc:288] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.744042 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a8cbbafd86402a9a3bf9e730b60c0e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:33.744051 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a8cbbafd86402a9a3bf9e730b60c0e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:33.744194 845 raft_consensus.cc:2454] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:33.744187 713 raft_consensus.cc:2454] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:33.744467 911 leader_election.cc:302] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.744622 1582 raft_consensus.cc:2781] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.744673 1582 raft_consensus.cc:482] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.744697 1582 raft_consensus.cc:3037] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.745478 1582 raft_consensus.cc:504] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.745791 1582 leader_election.cc:288] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:33.746194 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a8cbbafd86402a9a3bf9e730b60c0e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:33.746306 713 raft_consensus.cc:3037] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.746346 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a8cbbafd86402a9a3bf9e730b60c0e" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:33.746503 845 raft_consensus.cc:3037] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.747143 713 raft_consensus.cc:2454] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:33.747364 845 raft_consensus.cc:2454] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:33.747429 911 leader_election.cc:302] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:33.747615 1582 raft_consensus.cc:2781] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.747702 1582 raft_consensus.cc:686] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:33.747828 1582 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:33.749492 436 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:33.889367 1670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:33.889607 1670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:33.892726 1670 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:58:33.985951 713 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.986196 845 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.986459 1582 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.986868 1582 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.987844 583 raft_consensus.cc:1261] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.988427 844 raft_consensus.cc:3037] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.988478 1573 consensus_queue.cc:1035] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.989486 844 raft_consensus.cc:1261] T f42fb6ac38fc4b4782814cae0891b0cc P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.989862 1573 consensus_queue.cc:1035] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.995780 845 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.996199 1667 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.997927 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:33.998291 1667 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.743 ms
time per row: 0.0623713 ms
I20240503 07:58:34.060842 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41884:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:34.066088 956 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6c2af6b9764ad1b464973bc96853ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.066099 561 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6c2af6b9764ad1b464973bc96853ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.066393 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6c2af6b9764ad1b464973bc96853ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.066392 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6c2af6b9764ad1b464973bc96853ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.066519 955 tablet_service.cc:1467] Processing CreateTablet for tablet ac39a827f881410c83898fe3a26e34f1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.066632 694 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6c2af6b9764ad1b464973bc96853ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.066751 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac39a827f881410c83898fe3a26e34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.066852 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6c2af6b9764ad1b464973bc96853ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.068815 1731 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:34.069397 955 tablet_service.cc:1467] Processing CreateTablet for tablet 611c37485cd24fae9374586252925abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.069607 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.069684 561 tablet_service.cc:1467] Processing CreateTablet for tablet ac39a827f881410c83898fe3a26e34f1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.069684 812 tablet_service.cc:1467] Processing CreateTablet for tablet ac39a827f881410c83898fe3a26e34f1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.069795 1730 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:34.069909 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac39a827f881410c83898fe3a26e34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.069911 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac39a827f881410c83898fe3a26e34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.070601 1730 tablet_bootstrap.cc:654] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.071168 563 tablet_service.cc:1467] Processing CreateTablet for tablet 611c37485cd24fae9374586252925abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.071362 1729 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:34.071374 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.071916 1731 tablet_bootstrap.cc:654] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.071960 694 tablet_service.cc:1467] Processing CreateTablet for tablet 611c37485cd24fae9374586252925abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.072175 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.072988 1731 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:34.073077 1731 ts_tablet_manager.cc:1367] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:58:34.073550 1731 raft_consensus.cc:348] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.073662 1731 raft_consensus.cc:374] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.073704 1731 raft_consensus.cc:729] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.073807 1731 consensus_queue.cc:260] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.074151 1731 ts_tablet_manager.cc:1398] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.074303 1730 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:34.074349 1731 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:34.074378 1730 ts_tablet_manager.cc:1367] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:58:34.074772 1734 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:34.074796 1730 raft_consensus.cc:348] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.074894 1730 raft_consensus.cc:374] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.074968 1730 raft_consensus.cc:729] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.075091 1730 consensus_queue.cc:260] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.075171 1731 tablet_bootstrap.cc:654] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.076090 1730 ts_tablet_manager.cc:1398] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:34.076237 1730 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:34.076424 1734 tablet_bootstrap.cc:654] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.077469 1731 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:34.077545 1731 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:58:34.077569 1730 tablet_bootstrap.cc:654] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.077924 1731 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.078022 1731 raft_consensus.cc:374] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.078053 1731 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.078169 1731 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.078495 1731 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.079285 1734 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:34.079352 1730 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:34.079367 1734 ts_tablet_manager.cc:1367] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:58:34.079442 1730 ts_tablet_manager.cc:1367] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:58:34.079774 1734 raft_consensus.cc:348] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.079864 1734 raft_consensus.cc:374] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.079892 1734 raft_consensus.cc:729] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.079890 1730 raft_consensus.cc:348] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.080022 1730 raft_consensus.cc:374] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.080065 1730 raft_consensus.cc:729] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.080027 1734 consensus_queue.cc:260] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.080204 1730 consensus_queue.cc:260] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.080302 1734 ts_tablet_manager.cc:1398] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.080530 1730 ts_tablet_manager.cc:1398] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:58:34.080693 1730 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:34.081490 1730 tablet_bootstrap.cc:654] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.078888 1729 tablet_bootstrap.cc:654] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.083014 1729 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:34.083112 1729 ts_tablet_manager.cc:1367] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:58:34.083469 1729 raft_consensus.cc:348] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.083573 1729 raft_consensus.cc:374] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.083614 1729 raft_consensus.cc:729] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.083760 1729 consensus_queue.cc:260] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.084003 1729 ts_tablet_manager.cc:1398] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:34.084167 1729 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:34.084918 1729 tablet_bootstrap.cc:654] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.085522 1730 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:34.085605 1730 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:58:34.085803 1729 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:34.085880 1729 ts_tablet_manager.cc:1367] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:34.086045 1730 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.086125 1730 raft_consensus.cc:374] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.086158 1730 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.086285 1730 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.086278 1729 raft_consensus.cc:348] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.086373 1729 raft_consensus.cc:374] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.086412 1729 raft_consensus.cc:729] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.086530 1729 consensus_queue.cc:260] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.086588 1730 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:58:34.086854 1729 ts_tablet_manager.cc:1398] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.087064 1729 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:34.087811 1729 tablet_bootstrap.cc:654] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.088935 1729 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:34.089004 1729 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:34.089283 1729 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.089354 1729 raft_consensus.cc:374] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.089373 1729 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.089455 1729 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.089653 1729 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:34.107816 1728 raft_consensus.cc:482] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.107934 1728 raft_consensus.cc:504] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.108355 1728 leader_election.cc:288] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:34.108774 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac39a827f881410c83898fe3a26e34f1" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:34.108933 845 raft_consensus.cc:2454] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.109295 911 leader_election.cc:302] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.109484 1728 raft_consensus.cc:2781] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.109540 1728 raft_consensus.cc:482] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.109582 1728 raft_consensus.cc:3037] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.110464 1728 raft_consensus.cc:504] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.110795 1728 leader_election.cc:288] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:34.111155 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac39a827f881410c83898fe3a26e34f1" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:34.111280 845 raft_consensus.cc:3037] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.112221 845 raft_consensus.cc:2454] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:34.112516 911 leader_election.cc:302] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.112572 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac39a827f881410c83898fe3a26e34f1" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:34.112625 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac39a827f881410c83898fe3a26e34f1" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:34.112692 1728 raft_consensus.cc:2781] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.112737 583 raft_consensus.cc:2454] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.112766 1728 raft_consensus.cc:686] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:34.112901 1728 consensus_queue.cc:237] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.114522 432 catalog_manager.cc:5484] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:34.309370 1728 raft_consensus.cc:482] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.309504 1728 raft_consensus.cc:504] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.309878 1728 leader_election.cc:288] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.310355 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:34.310355 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:34.310509 713 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.310508 582 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.310817 911 leader_election.cc:302] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.311021 1728 raft_consensus.cc:2781] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.311085 1728 raft_consensus.cc:482] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.311139 1728 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.312014 1728 raft_consensus.cc:504] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.312355 1728 leader_election.cc:288] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.312692 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:34.312737 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:34.312820 582 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.312827 713 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.313730 713 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:34.313730 582 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:34.314090 911 leader_election.cc:302] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.314275 1728 raft_consensus.cc:2781] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.314378 1728 raft_consensus.cc:686] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:34.314513 1728 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:34.316143 432 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:34.361950 1578 raft_consensus.cc:482] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.362098 1578 raft_consensus.cc:504] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.362596 1578 leader_election.cc:288] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.363126 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:34.363126 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:34.363277 713 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 0.
I20240503 07:58:34.363277 976 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 0.
I20240503 07:58:34.363579 518 leader_election.cc:302] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:34.363775 1578 raft_consensus.cc:2781] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.363831 1578 raft_consensus.cc:482] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.363868 1578 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.364805 1578 raft_consensus.cc:504] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.365131 1578 leader_election.cc:288] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Requested vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.365469 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:34.365468 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "27d51a644d914549b3dfb3f104d396e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:34.365595 713 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.365595 976 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.366547 713 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:34.366547 976 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d51a644d914549b3dfb3f104d396e8 in term 1.
I20240503 07:58:34.366830 519 leader_election.cc:302] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.367053 1578 raft_consensus.cc:2781] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.367239 1578 raft_consensus.cc:686] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 LEADER]: Becoming Leader. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Running, Role: LEADER
I20240503 07:58:34.367375 1578 consensus_queue.cc:237] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.369100 432 catalog_manager.cc:5484] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "27d51a644d914549b3dfb3f104d396e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:34.543866 1747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:34.544111 1747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:34.547343 1747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:58:34.625411 976 raft_consensus.cc:1261] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.625458 713 raft_consensus.cc:1261] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 27d51a644d914549b3dfb3f104d396e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.625811 1745 consensus_queue.cc:1035] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:34.626329 1745 consensus_queue.cc:1035] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:34.629395 845 raft_consensus.cc:1261] T ac39a827f881410c83898fe3a26e34f1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.629685 582 raft_consensus.cc:3037] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.630728 582 raft_consensus.cc:1261] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.631091 1582 consensus_queue.cc:1035] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:34.631397 1582 consensus_queue.cc:1035] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:34.637240 583 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.637616 714 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:34.637591 1582 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:34.642659 1582 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.483 ms
time per row: 0.0517416 ms
I20240503 07:58:34.692364 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41884:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:34.698241 694 tablet_service.cc:1467] Processing CreateTablet for tablet 24e17467ff094742bf72f568c0953973 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.698239 563 tablet_service.cc:1467] Processing CreateTablet for tablet 24e17467ff094742bf72f568c0953973 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.698364 812 tablet_service.cc:1467] Processing CreateTablet for tablet 24e17467ff094742bf72f568c0953973 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.698554 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e17467ff094742bf72f568c0953973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.698554 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e17467ff094742bf72f568c0953973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.698647 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e17467ff094742bf72f568c0953973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.699205 956 tablet_service.cc:1467] Processing CreateTablet for tablet af4caa7f67b9473192241d4030f07f99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.699463 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4caa7f67b9473192241d4030f07f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.701016 955 tablet_service.cc:1467] Processing CreateTablet for tablet 71d661e6c1374f9282f8eff323861e1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.701336 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d661e6c1374f9282f8eff323861e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.701495 563 tablet_service.cc:1467] Processing CreateTablet for tablet af4caa7f67b9473192241d4030f07f99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.701707 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4caa7f67b9473192241d4030f07f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.701867 825 tablet_service.cc:1467] Processing CreateTablet for tablet af4caa7f67b9473192241d4030f07f99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.701999 812 tablet_service.cc:1467] Processing CreateTablet for tablet 71d661e6c1374f9282f8eff323861e1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.702104 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4caa7f67b9473192241d4030f07f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.702214 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d661e6c1374f9282f8eff323861e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.703976 1808 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:34.704141 1807 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:34.704802 694 tablet_service.cc:1467] Processing CreateTablet for tablet 71d661e6c1374f9282f8eff323861e1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:34.704883 1808 tablet_bootstrap.cc:654] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.705003 1807 tablet_bootstrap.cc:654] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.705181 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d661e6c1374f9282f8eff323861e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:34.705636 1806 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:34.706153 1807 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:34.706228 1807 ts_tablet_manager.cc:1367] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:34.706533 1806 tablet_bootstrap.cc:654] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.706686 1807 raft_consensus.cc:348] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.706799 1807 raft_consensus.cc:374] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.706843 1807 raft_consensus.cc:729] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.707082 1807 consensus_queue.cc:260] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.707360 1807 ts_tablet_manager.cc:1398] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:34.707521 1807 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:34.708029 1806 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:34.708112 1806 ts_tablet_manager.cc:1367] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:34.708211 1805 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:34.708248 1807 tablet_bootstrap.cc:654] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.708626 1806 raft_consensus.cc:348] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.708732 1806 raft_consensus.cc:374] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.708763 1806 raft_consensus.cc:729] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.708875 1806 consensus_queue.cc:260] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.709123 1806 ts_tablet_manager.cc:1398] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.709303 1806 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:34.709304 1805 tablet_bootstrap.cc:654] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.710551 1806 tablet_bootstrap.cc:654] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.710577 1805 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:34.710675 1805 ts_tablet_manager.cc:1367] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:34.711587 1808 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:34.711702 1808 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:58:34.711939 1806 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:34.712011 1806 ts_tablet_manager.cc:1367] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:34.712023 1807 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:34.712098 1807 ts_tablet_manager.cc:1367] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:58:34.712215 1808 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.712334 1808 raft_consensus.cc:374] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.712378 1808 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.712388 1806 raft_consensus.cc:348] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.712487 1806 raft_consensus.cc:374] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.712517 1806 raft_consensus.cc:729] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.712510 1808 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.712626 1806 consensus_queue.cc:260] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.712561 1807 raft_consensus.cc:348] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.712705 1807 raft_consensus.cc:374] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.712749 1807 raft_consensus.cc:729] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.712806 1808 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:34.712870 1806 ts_tablet_manager.cc:1398] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:34.713006 1808 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:34.712922 1807 consensus_queue.cc:260] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.713254 1807 ts_tablet_manager.cc:1398] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:34.713428 1807 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:34.714180 1805 raft_consensus.cc:348] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.714367 1805 raft_consensus.cc:374] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.714448 1805 raft_consensus.cc:729] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.714460 1808 tablet_bootstrap.cc:654] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.714648 1805 consensus_queue.cc:260] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.715029 1805 ts_tablet_manager.cc:1398] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:58:34.715245 1805 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:34.715579 1808 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:34.715664 1808 ts_tablet_manager.cc:1367] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:58:34.716109 1808 raft_consensus.cc:348] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.716221 1808 raft_consensus.cc:374] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.716264 1808 raft_consensus.cc:729] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.716378 1808 consensus_queue.cc:260] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.716648 1808 ts_tablet_manager.cc:1398] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:34.717339 1805 tablet_bootstrap.cc:654] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.717813 1807 tablet_bootstrap.cc:654] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:34.718964 1807 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:34.719041 1807 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:58:34.719400 1807 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.719480 1807 raft_consensus.cc:374] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.719511 1807 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.719619 1807 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.719851 1807 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:34.720484 1805 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:34.720602 1805 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:58:34.721055 1805 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.721138 1805 raft_consensus.cc:374] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:34.721165 1805 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:34.721302 1805 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.721884 1805 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:34.723469 1579 raft_consensus.cc:482] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.723551 1579 raft_consensus.cc:504] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.723904 1579 leader_election.cc:288] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:34.724249 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e17467ff094742bf72f568c0953973" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:34.724300 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e17467ff094742bf72f568c0953973" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:34.724359 582 raft_consensus.cc:2454] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:34.724419 845 raft_consensus.cc:2454] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:34.724727 649 leader_election.cc:302] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:34.724889 1579 raft_consensus.cc:2781] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.724946 1579 raft_consensus.cc:482] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.724988 1579 raft_consensus.cc:3037] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.725745 1579 raft_consensus.cc:504] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.726051 1579 leader_election.cc:288] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:34.726303 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e17467ff094742bf72f568c0953973" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:34.726390 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e17467ff094742bf72f568c0953973" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:34.726430 582 raft_consensus.cc:3037] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.726485 845 raft_consensus.cc:3037] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.727324 845 raft_consensus.cc:2454] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:34.727324 582 raft_consensus.cc:2454] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:34.727587 649 leader_election.cc:302] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:34.727730 1579 raft_consensus.cc:2781] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.727813 1579 raft_consensus.cc:686] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:34.727941 1579 consensus_queue.cc:237] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.729343 441 catalog_manager.cc:5484] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:34.776028 1801 raft_consensus.cc:482] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.776152 1801 raft_consensus.cc:504] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.776587 1801 leader_election.cc:288] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:34.777081 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4caa7f67b9473192241d4030f07f99" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:34.777172 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4caa7f67b9473192241d4030f07f99" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:34.777256 582 raft_consensus.cc:2454] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.777297 845 raft_consensus.cc:2454] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:34.777606 911 leader_election.cc:302] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.777782 1801 raft_consensus.cc:2781] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.777845 1801 raft_consensus.cc:482] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.777889 1801 raft_consensus.cc:3037] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.778744 1801 raft_consensus.cc:504] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.779125 1801 leader_election.cc:288] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:34.779438 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4caa7f67b9473192241d4030f07f99" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:34.779518 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4caa7f67b9473192241d4030f07f99" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:34.779563 582 raft_consensus.cc:3037] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.779631 845 raft_consensus.cc:3037] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.780468 845 raft_consensus.cc:2454] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:34.780472 582 raft_consensus.cc:2454] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:34.780722 911 leader_election.cc:302] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.780902 1801 raft_consensus.cc:2781] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.780997 1801 raft_consensus.cc:686] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:34.781137 1801 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:34.782755 441 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:34.867005 1784 raft_consensus.cc:482] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:34.867152 1784 raft_consensus.cc:504] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.868150 1784 leader_election.cc:288] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.871803 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d661e6c1374f9282f8eff323861e1e" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:34.871842 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d661e6c1374f9282f8eff323861e1e" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:34.871949 976 raft_consensus.cc:2454] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 0.
I20240503 07:58:34.871980 713 raft_consensus.cc:2454] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 0.
I20240503 07:58:34.872376 780 leader_election.cc:302] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:34.872613 1784 raft_consensus.cc:2781] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:34.872670 1784 raft_consensus.cc:482] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:34.872705 1784 raft_consensus.cc:3037] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.873560 1784 raft_consensus.cc:504] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.873880 1784 leader_election.cc:288] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 election: Requested vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:34.874266 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d661e6c1374f9282f8eff323861e1e" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:34.874289 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d661e6c1374f9282f8eff323861e1e" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:34.874374 976 raft_consensus.cc:3037] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.874378 713 raft_consensus.cc:3037] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:34.875110 713 raft_consensus.cc:2454] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 1.
I20240503 07:58:34.875118 976 raft_consensus.cc:2454] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 1.
I20240503 07:58:34.875412 781 leader_election.cc:302] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:34.875581 1784 raft_consensus.cc:2781] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:34.875766 1784 raft_consensus.cc:686] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Becoming Leader. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Running, Role: LEADER
I20240503 07:58:34.876068 1784 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:34.878284 441 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:35.006567 1825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.006816 1825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.009843 1825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:58:35.094494 845 raft_consensus.cc:1261] T 24e17467ff094742bf72f568c0953973 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.095368 1579 consensus_queue.cc:1035] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.096151 582 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.097112 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.097173 1801 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.097553 976 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.097625 1801 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.097980 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.107267 1823 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.111804 1823 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.111894 582 raft_consensus.cc:1261] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.112622 1792 consensus_queue.cc:1035] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.89 ms
time per row: 0.0534449 ms
I20240503 07:58:35.164420 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41884:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:35.170053 694 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0a00c4bec748bc957283814ca26ae1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.170398 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0a00c4bec748bc957283814ca26ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.170590 561 tablet_service.cc:1467] Processing CreateTablet for tablet 6e96426aaa194d8cb60e03a31abfd83d (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.170701 563 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0a00c4bec748bc957283814ca26ae1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.170830 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e96426aaa194d8cb60e03a31abfd83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.170943 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0a00c4bec748bc957283814ca26ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.171466 825 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0a00c4bec748bc957283814ca26ae1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.171710 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0a00c4bec748bc957283814ca26ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.172713 1805 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:35.172926 693 tablet_service.cc:1467] Processing CreateTablet for tablet 6e96426aaa194d8cb60e03a31abfd83d (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.172971 812 tablet_service.cc:1467] Processing CreateTablet for tablet 2dac1e203d8a47ae8cbc331ffe5f211b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.173147 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e96426aaa194d8cb60e03a31abfd83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.173185 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dac1e203d8a47ae8cbc331ffe5f211b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.173740 1805 tablet_bootstrap.cc:654] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.174129 955 tablet_service.cc:1467] Processing CreateTablet for tablet 6e96426aaa194d8cb60e03a31abfd83d (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.174360 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e96426aaa194d8cb60e03a31abfd83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.174871 1806 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:35.175184 956 tablet_service.cc:1467] Processing CreateTablet for tablet 2dac1e203d8a47ae8cbc331ffe5f211b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.175393 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dac1e203d8a47ae8cbc331ffe5f211b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.175630 1806 tablet_bootstrap.cc:654] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.176225 1808 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:35.176537 694 tablet_service.cc:1467] Processing CreateTablet for tablet 2dac1e203d8a47ae8cbc331ffe5f211b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:35.176733 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dac1e203d8a47ae8cbc331ffe5f211b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.176951 1807 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:35.177014 1808 tablet_bootstrap.cc:654] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.177618 1807 tablet_bootstrap.cc:654] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.178038 1808 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:35.178129 1808 ts_tablet_manager.cc:1367] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:35.178511 1807 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:35.178577 1807 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:35.178539 1808 raft_consensus.cc:348] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.178637 1808 raft_consensus.cc:374] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.178666 1808 raft_consensus.cc:729] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.178777 1808 consensus_queue.cc:260] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.179008 1807 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.179108 1807 raft_consensus.cc:374] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.179108 1808 ts_tablet_manager.cc:1398] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:35.179137 1807 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.179266 1808 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Bootstrap starting.
I20240503 07:58:35.179265 1806 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:35.179250 1807 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.179348 1806 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:58:35.179487 1807 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:35.179795 1806 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.179888 1806 raft_consensus.cc:374] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.179919 1806 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.180040 1806 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.180202 1807 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Bootstrap starting.
I20240503 07:58:35.180294 1806 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:35.180456 1806 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:35.180930 1807 tablet_bootstrap.cc:654] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.181274 1806 tablet_bootstrap.cc:654] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.181847 1807 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: No bootstrap required, opened a new log
I20240503 07:58:35.181870 1805 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:35.181913 1807 ts_tablet_manager.cc:1367] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:35.181941 1805 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:58:35.182351 1807 raft_consensus.cc:348] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.182358 1805 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.182442 1807 raft_consensus.cc:374] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.182442 1805 raft_consensus.cc:374] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.182471 1807 raft_consensus.cc:729] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.182493 1805 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.182582 1807 consensus_queue.cc:260] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.182600 1805 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.182823 1807 ts_tablet_manager.cc:1398] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:35.182835 1805 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:35.183140 1805 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:58:35.183908 1805 tablet_bootstrap.cc:654] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.186496 1808 tablet_bootstrap.cc:654] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.187566 1808 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: No bootstrap required, opened a new log
I20240503 07:58:35.187674 1805 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: No bootstrap required, opened a new log
I20240503 07:58:35.187744 1805 ts_tablet_manager.cc:1367] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:58:35.187695 1808 ts_tablet_manager.cc:1367] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:58:35.187923 1806 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:35.188071 1806 ts_tablet_manager.cc:1367] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:58:35.188058 1805 raft_consensus.cc:348] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.188130 1805 raft_consensus.cc:374] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.188155 1805 raft_consensus.cc:729] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.188246 1805 consensus_queue.cc:260] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.188390 1808 raft_consensus.cc:348] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.188478 1805 ts_tablet_manager.cc:1398] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:35.188467 1806 raft_consensus.cc:348] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.188566 1806 raft_consensus.cc:374] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.188598 1808 raft_consensus.cc:374] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.188619 1806 raft_consensus.cc:729] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.188719 1808 raft_consensus.cc:729] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.188781 1806 consensus_queue.cc:260] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.188905 1808 consensus_queue.cc:260] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.189124 1806 ts_tablet_manager.cc:1398] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:35.189285 1806 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3: Bootstrap starting.
I20240503 07:58:35.189407 1808 ts_tablet_manager.cc:1398] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:35.190086 1806 tablet_bootstrap.cc:654] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.191689 1806 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3: No bootstrap required, opened a new log
I20240503 07:58:35.191771 1806 ts_tablet_manager.cc:1367] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:58:35.192169 1806 raft_consensus.cc:348] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.192284 1806 raft_consensus.cc:374] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.192315 1806 raft_consensus.cc:729] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.192438 1806 consensus_queue.cc:260] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.192698 1806 ts_tablet_manager.cc:1398] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:35.270661 1801 raft_consensus.cc:482] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:35.270787 1801 raft_consensus.cc:504] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.271209 1801 leader_election.cc:288] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:35.271559 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dac1e203d8a47ae8cbc331ffe5f211b" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:35.271572 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dac1e203d8a47ae8cbc331ffe5f211b" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:35.271680 713 raft_consensus.cc:2454] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:35.271699 843 raft_consensus.cc:2454] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 0.
I20240503 07:58:35.271914 911 leader_election.cc:302] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:35.272063 1801 raft_consensus.cc:2781] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:35.272117 1801 raft_consensus.cc:482] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:35.272142 1801 raft_consensus.cc:3037] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.272949 1801 raft_consensus.cc:504] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.273238 1801 leader_election.cc:288] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Requested vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:35.273762 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dac1e203d8a47ae8cbc331ffe5f211b" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:35.273772 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dac1e203d8a47ae8cbc331ffe5f211b" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:35.273864 843 raft_consensus.cc:3037] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.273870 713 raft_consensus.cc:3037] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.274649 843 raft_consensus.cc:2454] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:35.274667 713 raft_consensus.cc:2454] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:35.274971 911 leader_election.cc:302] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:58:35.275161 1801 raft_consensus.cc:2781] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:35.275272 1801 raft_consensus.cc:686] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:35.275413 1801 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:35.276960 432 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 0 to 1, leader changed from <none> to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:35.289642 1862 raft_consensus.cc:482] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:35.289765 1862 raft_consensus.cc:504] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.290314 1862 leader_election.cc:288] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:35.290721 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0a00c4bec748bc957283814ca26ae1" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:35.290855 713 raft_consensus.cc:2454] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 0.
I20240503 07:58:35.291200 780 leader_election.cc:302] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:35.291383 1862 raft_consensus.cc:2781] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:35.291440 1862 raft_consensus.cc:482] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:35.291484 1862 raft_consensus.cc:3037] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.292450 1862 raft_consensus.cc:504] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.292780 1862 leader_election.cc:288] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:35.293090 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0a00c4bec748bc957283814ca26ae1" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:35.293208 713 raft_consensus.cc:3037] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.294073 713 raft_consensus.cc:2454] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 1.
I20240503 07:58:35.294335 780 leader_election.cc:302] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:35.294411 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0a00c4bec748bc957283814ca26ae1" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:35.294489 1862 raft_consensus.cc:2781] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:35.294471 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0a00c4bec748bc957283814ca26ae1" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:35.294555 1862 raft_consensus.cc:686] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Becoming Leader. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Running, Role: LEADER
I20240503 07:58:35.294560 582 raft_consensus.cc:2454] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 0.
I20240503 07:58:35.294682 1862 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.296310 432 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:35.400419 1792 raft_consensus.cc:482] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:35.400566 1792 raft_consensus.cc:504] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.400993 1792 leader_election.cc:288] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:35.401481 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e96426aaa194d8cb60e03a31abfd83d" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8" is_pre_election: true
I20240503 07:58:35.401479 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e96426aaa194d8cb60e03a31abfd83d" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:35.401623 976 raft_consensus.cc:2454] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:35.401628 582 raft_consensus.cc:2454] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 0.
I20240503 07:58:35.401963 649 leader_election.cc:302] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:35.402153 1792 raft_consensus.cc:2781] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:35.402204 1792 raft_consensus.cc:482] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:35.402251 1792 raft_consensus.cc:3037] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.403350 1792 raft_consensus.cc:504] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.403702 1792 leader_election.cc:288] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:35.404004 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e96426aaa194d8cb60e03a31abfd83d" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:35.404114 976 raft_consensus.cc:3037] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.404215 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e96426aaa194d8cb60e03a31abfd83d" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d51a644d914549b3dfb3f104d396e8"
I20240503 07:58:35.404333 582 raft_consensus.cc:3037] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.405030 976 raft_consensus.cc:2454] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:35.405103 582 raft_consensus.cc:2454] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:35.405411 649 leader_election.cc:302] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:35.405609 1792 raft_consensus.cc:2781] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:35.405695 1792 raft_consensus.cc:686] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:35.405825 1792 consensus_queue.cc:237] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:58:35.407483 432 catalog_manager.cc:5484] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:35.532073 1878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.532363 1878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.535527 1878 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:58:35.635092 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.635092 582 raft_consensus.cc:3037] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.635818 1862 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.636271 582 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.637663 1862 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.638585 714 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.639180 1801 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.641738 843 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.643476 1801 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.643595 582 raft_consensus.cc:1261] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.647117 976 raft_consensus.cc:1261] T 6e96426aaa194d8cb60e03a31abfd83d P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:35.652546 1792 consensus_queue.cc:1035] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.652894 1792 consensus_queue.cc:1035] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:35.673733 1854 mvcc.cc:205] Tried to move back new op lower bound from 7023505881670955008 to 7023505880169291776. Current Snapshot: MvccSnapshot[applied={T|T < 7023505881646039040 or (T in {7023505881670955008})}]
I20240503 07:58:35.674471 1912 mvcc.cc:205] Tried to move back new op lower bound from 7023505881628913664 to 7023505880248033280. Current Snapshot: MvccSnapshot[applied={T|T < 7023505881628913664 or (T in {7023505881628913664})}]
I20240503 07:58:35.685438 1923 mvcc.cc:205] Tried to move back new op lower bound from 7023505881680896000 to 7023505880703389696. Current Snapshot: MvccSnapshot[applied={T|T < 7023505881656205312}]
I20240503 07:58:35.690739 1776 mvcc.cc:205] Tried to move back new op lower bound from 7023505881680896000 to 7023505880703389696. Current Snapshot: MvccSnapshot[applied={T|T < 7023505881656205312}]
INSERT report
rows total: 2000
time total: 125.598 ms
time per row: 0.0627988 ms
I20240503 07:58:35.702021 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:33799
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:37543
--webserver_port=42073
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:58:35.810367 1930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.810577 1930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:35.810633 1930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.813634 1930 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:35.813694 1930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:35.813789 1930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:35.817332 1930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:33799
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=42073
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.1930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:35.818162 1930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:35.819355 1930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:35.828332 1935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.828701 1938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.829362 1936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:35.830040 1930 server_base.cc:1034] running on GCE node
I20240503 07:58:35.830397 1930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:35.831009 1930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:35.832178 1930 hybrid_clock.cc:648] HybridClock initialized: now 1714723115832169 us; error 38 us; skew 500 ppm
I20240503 07:58:35.834224 1930 webserver.cc:460] Webserver started at http://127.0.99.133:42073/ using document root <none> and password file <none>
I20240503 07:58:35.834830 1930 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:35.834904 1930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:35.838725 1930 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:35.840803 1944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.841886 1930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:35.841992 1930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9e85f3485213464fb71cb81d65aedbad"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:35.842475 1930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:35.869304 1930 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:35.869961 1930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:35.870126 1930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:35.870657 1930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:35.871608 1930 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:35.871659 1930 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.871703 1930 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:35.871727 1930 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.880204 1930 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33799
I20240503 07:58:35.880244 2057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33799 every 8 connection(s)
I20240503 07:58:35.881374 1930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:35.887497 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 1930
I20240503 07:58:35.887866 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:34213
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:37543
--webserver_port=44103
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:35.889979 2058 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:35.890218 2058 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:35.890949 2058 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:35.892102 432 ts_manager.cc:194] Re-registered known tserver with Master: 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799)
I20240503 07:58:35.892681 432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:47309
W20240503 07:58:35.997256 2061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.997473 2061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:35.997524 2061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.000615 2061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.000681 2061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.000766 2061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:36.004571 2061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:34213
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=44103
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.2061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.005498 2061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.006680 2061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:36.016088 2066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.016587 2067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.017210 2069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.017788 2061 server_base.cc:1034] running on GCE node
I20240503 07:58:36.018218 2061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.019023 2061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.020185 2061 hybrid_clock.cc:648] HybridClock initialized: now 1714723116020159 us; error 37 us; skew 500 ppm
I20240503 07:58:36.022293 2061 webserver.cc:460] Webserver started at http://127.0.99.134:44103/ using document root <none> and password file <none>
I20240503 07:58:36.022948 2061 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.023029 2061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.027036 2061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:36.029220 2075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.030344 2061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:36.030457 2061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:36.030990 2061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.059772 2061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.060386 2061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.060529 2061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.061015 2061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.061833 2061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.061878 2061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.061914 2061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.061933 2061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.070003 2061 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34213
I20240503 07:58:36.070032 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34213 every 8 connection(s)
I20240503 07:58:36.071007 2061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:36.073865 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 2061
I20240503 07:58:36.074215 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:35843
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:37543
--webserver_port=41273
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:36.079525 2189 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:36.079767 2189 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.080437 2189 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:36.081427 432 ts_manager.cc:194] Re-registered known tserver with Master: 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213)
I20240503 07:58:36.081940 432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:60795
W20240503 07:58:36.187728 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.187961 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:36.188017 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.191138 2192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.191202 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.191279 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:36.195320 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:35843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41273
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.196256 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.197345 2192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:36.206884 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.207448 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.208140 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.208783 2192 server_base.cc:1034] running on GCE node
I20240503 07:58:36.209210 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.209985 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.211197 2192 hybrid_clock.cc:648] HybridClock initialized: now 1714723116211164 us; error 56 us; skew 500 ppm
I20240503 07:58:36.213303 2192 webserver.cc:460] Webserver started at http://127.0.99.135:41273/ using document root <none> and password file <none>
I20240503 07:58:36.213927 2192 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.214008 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.217564 2192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:36.219863 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.220888 2192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 07:58:36.220999 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b8f135eb22964d4a8d8300956bce8017"
format_stamp: "Formatted at 2024-05-03 07:58:28 on dist-test-slave-11df"
I20240503 07:58:36.221408 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.239785 2192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.240432 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.240586 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.241088 2192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.241955 2192 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.242000 2192 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.242049 2192 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.242074 2192 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.250842 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35843
I20240503 07:58:36.250869 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35843 every 8 connection(s)
I20240503 07:58:36.251931 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:36.260351 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 2192
I20240503 07:58:36.260761 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:45427
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:37543
--webserver_port=41891
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:58:36.260798 2320 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:36.261004 2320 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.261503 2320 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:36.262445 432 ts_manager.cc:194] Re-registered known tserver with Master: b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843)
I20240503 07:58:36.263025 432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:48647
W20240503 07:58:36.370697 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.370982 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:36.371060 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.374231 2323 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.374293 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.374361 2323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:36.378098 2323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:45427
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=41891
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.2323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.378990 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.380090 2323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:36.389088 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.389689 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.390277 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.390900 2323 server_base.cc:1034] running on GCE node
I20240503 07:58:36.391311 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.391940 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.393083 2323 hybrid_clock.cc:648] HybridClock initialized: now 1714723116393053 us; error 48 us; skew 500 ppm
I20240503 07:58:36.394858 2323 webserver.cc:460] Webserver started at http://127.0.99.136:41891/ using document root <none> and password file <none>
I20240503 07:58:36.395490 2323 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.395574 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.398974 2323 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240503 07:58:36.400869 2337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.401952 2323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:36.402053 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "164b774d3a3346bca3572f022142e17c"
format_stamp: "Formatted at 2024-05-03 07:58:29 on dist-test-slave-11df"
I20240503 07:58:36.402554 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.420828 2323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.421444 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.421590 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.422086 2323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.422986 2323 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.423030 2323 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.423084 2323 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.423105 2323 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.432060 2323 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:45427
I20240503 07:58:36.432080 2450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:45427 every 8 connection(s)
I20240503 07:58:36.433061 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:36.436092 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 2323
I20240503 07:58:36.442111 2451 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:58:36.442305 2451 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.442802 2451 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:58:36.443905 432 ts_manager.cc:194] Re-registered known tserver with Master: 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427)
I20240503 07:58:36.444563 432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:53527
I20240503 07:58:36.628618 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:36.631446 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.632195 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.637341 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:36.640430 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:36.653331 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:36.656611 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.672295 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.839084 438 ts_manager.cc:292] Set tserver state for 27d51a644d914549b3dfb3f104d396e8 to MAINTENANCE_MODE
I20240503 07:58:36.895195 2058 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
W20240503 07:58:36.952739 2515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.952956 2515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.956018 2515 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:58:37.057096 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.057653 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.059893 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.062110 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.069850 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.073791 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.083880 2189 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:58:37.085345 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.089264 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.098956 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.100466 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.110231 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.113744 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.123684 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.123739 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.124447 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.135303 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.265322 2320 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:58:37.267853 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.273458 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.275565 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.276049 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.286257 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.286623 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.296466 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.299177 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:58:37.348979 2606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:37.349224 2606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:37.352325 2606 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:58:37.413434 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.415596 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.425408 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.426680 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.430428 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.431674 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.435119 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.443723 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.451117 2451 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:58:37.490137 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.491247 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.492748 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.495154 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.503602 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.506850 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.520902 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.540800 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.546419 714 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } }
I20240503 07:58:37.547742 582 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:37.547971 976 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:37.548200 1914 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:37.548471 2713 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:37.550670 1914 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:37.551709 582 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:37.552239 976 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:37.554790 441 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:37.559360 714 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:37.560714 976 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:37.561079 582 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:37.561110 1914 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:37.561424 2713 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:37.562793 2713 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:37.564199 427 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 07:58:37.564679 976 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:37.565053 582 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:37.567263 441 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
W20240503 07:58:37.575628 649 consensus_peers.cc:485] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843): Couldn't send request to peer b8f135eb22964d4a8d8300956bce8017. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eca5e6693b24d0e8f3d9c37a62a1bca. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:37.696043 2726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:37.696290 2726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:37.699323 2726 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:58:37.778132 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.778239 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.779428 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:37.780993 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:37.792306 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:37.792795 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.798645 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:37.808053 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:58:37.967820 2768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:37.968066 2768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:37.971138 2768 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:58:38.050911 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.053671 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:38.054610 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:38.058699 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:38.058717 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:38.066358 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.077502 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.083381 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.125830 2810 ts_tablet_manager.cc:902] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Initiating tablet copy from peer 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:38.126488 2810 tablet_copy_client.cc:308] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38831
I20240503 07:58:38.129925 734 tablet_copy_service.cc:140] P 7a36d94f14d04d74b632bc75d1779bc3: Received BeginTabletCopySession request for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca from peer b8f135eb22964d4a8d8300956bce8017 ({username='slave'} at 127.0.99.135:33891)
I20240503 07:58:38.130069 734 tablet_copy_service.cc:161] P 7a36d94f14d04d74b632bc75d1779bc3: Beginning new tablet copy session on tablet 1eca5e6693b24d0e8f3d9c37a62a1bca from peer b8f135eb22964d4a8d8300956bce8017 at {username='slave'} at 127.0.99.135:33891: session id = b8f135eb22964d4a8d8300956bce8017-1eca5e6693b24d0e8f3d9c37a62a1bca
I20240503 07:58:38.131259 734 tablet_copy_source_session.cc:205] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:38.134729 2810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:38.140373 2810 tablet_copy_client.cc:783] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:38.140558 2810 tablet_copy_client.cc:647] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:38.142040 2814 tablet_copy_client.cc:1002] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:38.250536 2816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.250785 2816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.254031 2816 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:58:38.337704 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:58:38.337690 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.341992 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.345088 561 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:38.345687 812 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:38.355090 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:58:38.363329 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:38.370383 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 07:58:38.406869 2513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 501
W20240503 07:58:38.425912 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:38.426720 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.477710 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:38.478540 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.480996 649 consensus_peers.cc:485] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.489050 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.541873 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:38.542629 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.606407 911 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.687695 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:38.855855 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:39.436082 2874 raft_consensus.cc:482] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.436218 2874 raft_consensus.cc:504] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:39.436662 2874 leader_election.cc:288] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:39.437117 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
W20240503 07:58:39.437687 649 leader_election.cc:334] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.437842 649 leader_election.cc:302] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3; no voters: 27d51a644d914549b3dfb3f104d396e8, 80bd2b9fafaf44bcb8c0f1b9f408841e
I20240503 07:58:39.437994 2874 raft_consensus.cc:2726] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:58:39.541944 2876 raft_consensus.cc:482] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.542076 2876 raft_consensus.cc:504] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:39.542491 2876 leader_election.cc:288] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:39.543231 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
W20240503 07:58:39.543675 911 leader_election.cc:334] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.543742 911 leader_election.cc:302] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters: 27d51a644d914549b3dfb3f104d396e8, 5cef4772b54342fe9d418f103c30fc33
I20240503 07:58:39.543860 2876 raft_consensus.cc:2726] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:58:39.568444 2876 raft_consensus.cc:482] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.568671 2876 raft_consensus.cc:504] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:39.569131 2876 leader_election.cc:288] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:39.569890 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
I20240503 07:58:39.570062 713 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 1.
I20240503 07:58:39.570377 911 leader_election.cc:302] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
W20240503 07:58:39.570528 911 leader_election.cc:334] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.570567 2876 raft_consensus.cc:2781] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:58:39.570637 2876 raft_consensus.cc:482] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.570681 2876 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:39.571571 2876 raft_consensus.cc:504] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:39.571883 2876 leader_election.cc:288] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:39.572384 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6c2af6b9764ad1b464973bc96853ec" candidate_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:39.572515 713 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:58:39.572872 911 leader_election.cc:334] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.573616 713 raft_consensus.cc:2454] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bd2b9fafaf44bcb8c0f1b9f408841e in term 2.
I20240503 07:58:39.573899 911 leader_election.cc:302] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters: 27d51a644d914549b3dfb3f104d396e8
I20240503 07:58:39.574040 2876 raft_consensus.cc:2781] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:39.574139 2876 raft_consensus.cc:686] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 LEADER]: Becoming Leader. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:58:39.574324 2876 consensus_queue.cc:237] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:58:39.576023 432 catalog_manager.cc:5484] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: term changed from 1 to 2, leader changed from 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) to 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132). New cstate: current_term: 2 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:39.596962 2884 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.597095 2884 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:39.597484 2884 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:58:39.597892 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3" is_pre_election: true
W20240503 07:58:39.598523 780 leader_election.cc:334] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.598583 780 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33; no voters: 27d51a644d914549b3dfb3f104d396e8, 7a36d94f14d04d74b632bc75d1779bc3
I20240503 07:58:39.598708 2884 raft_consensus.cc:2726] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:58:39.642477 2810 tablet_copy_client.cc:523] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:39.644702 2810 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Bootstrap starting.
I20240503 07:58:39.650156 2810 log.cc:824] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:39.665920 2724 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.666054 2724 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:39.666451 2724 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:39.667068 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33" is_pre_election: true
I20240503 07:58:39.667217 843 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 1.
I20240503 07:58:39.667541 649 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
W20240503 07:58:39.667712 649 leader_election.cc:334] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.667712 2724 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:58:39.667802 2724 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.667846 2724 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:39.668673 2724 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:39.668985 2724 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:39.669607 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:58:39.669729 843 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:58:39.669816 649 leader_election.cc:334] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.670627 843 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 2.
I20240503 07:58:39.670902 649 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters: 27d51a644d914549b3dfb3f104d396e8
I20240503 07:58:39.671080 2724 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:39.671173 2724 raft_consensus.cc:686] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:39.671315 2724 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:39.671911 2810 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:39.672240 2810 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Bootstrap complete.
I20240503 07:58:39.672574 2810 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.004s
I20240503 07:58:39.673385 432 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 1 to 2, leader changed from 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:39.675207 2810 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:39.675472 2810 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f135eb22964d4a8d8300956bce8017, State: Initialized, Role: LEARNER
I20240503 07:58:39.675834 2810 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:39.676566 2810 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 07:58:39.677188 734 tablet_copy_service.cc:342] P 7a36d94f14d04d74b632bc75d1779bc3: Request end of tablet copy session b8f135eb22964d4a8d8300956bce8017-1eca5e6693b24d0e8f3d9c37a62a1bca received from {username='slave'} at 127.0.99.135:33891
I20240503 07:58:39.677274 734 tablet_copy_service.cc:433] P 7a36d94f14d04d74b632bc75d1779bc3: ending tablet copy session b8f135eb22964d4a8d8300956bce8017-1eca5e6693b24d0e8f3d9c37a62a1bca on tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with peer b8f135eb22964d4a8d8300956bce8017
I20240503 07:58:39.783762 2884 raft_consensus.cc:482] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.783908 2884 raft_consensus.cc:504] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:39.784322 2884 leader_election.cc:288] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:39.784816 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" is_pre_election: true
I20240503 07:58:39.784972 976 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 1.
I20240503 07:58:39.785293 781 leader_election.cc:302] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
W20240503 07:58:39.785430 780 leader_election.cc:334] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.785465 2884 raft_consensus.cc:2781] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:58:39.785533 2884 raft_consensus.cc:482] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27d51a644d914549b3dfb3f104d396e8)
I20240503 07:58:39.785576 2884 raft_consensus.cc:3037] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:39.786371 2884 raft_consensus.cc:504] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:39.786667 2884 leader_election.cc:288] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 election: Requested vote from peers 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481), 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:39.787075 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97138ab5e3394c41b83382a22df0a198" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:58:39.787214 976 raft_consensus.cc:3037] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:58:39.787546 780 leader_election.cc:334] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:39.788079 976 raft_consensus.cc:2454] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 2.
I20240503 07:58:39.788323 781 leader_election.cc:302] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cef4772b54342fe9d418f103c30fc33, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters: 27d51a644d914549b3dfb3f104d396e8
I20240503 07:58:39.788460 2884 raft_consensus.cc:2781] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:39.788605 2884 raft_consensus.cc:686] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [term 2 LEADER]: Becoming Leader. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Running, Role: LEADER
I20240503 07:58:39.788733 2884 consensus_queue.cc:237] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:39.790280 432 catalog_manager.cc:5484] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: term changed from 1 to 2, leader changed from 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) to 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:39.857247 2274 raft_consensus.cc:1203] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 07:58:40.045835 713 raft_consensus.cc:1261] T 7f6c2af6b9764ad1b464973bc96853ec P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:58:40.046576 2877 consensus_queue.cc:1035] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:58:40.047844 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.110333 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:58:40.110771 2724 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:58:40.114634 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.216594 976 raft_consensus.cc:1261] T 97138ab5e3394c41b83382a22df0a198 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:58:40.217077 2884 consensus_queue.cc:1035] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:58:40.220568 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.428555 2724 raft_consensus.cc:1050] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: attempting to promote NON_VOTER b8f135eb22964d4a8d8300956bce8017 to VOTER
I20240503 07:58:40.428977 2724 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:40.430315 976 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:40.430622 2274 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:40.430822 2939 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:58:40.431138 649 consensus_peers.cc:485] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.431358 2724 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:40.433950 2939 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.434478 2274 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.435781 976 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.436471 432 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 5 to 6, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:40.440261 713 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:40.441309 976 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:40.441428 2274 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:40.441650 2874 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:40.442325 2874 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:40.443344 2929 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.443744 976 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.444799 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:40.445358 2274 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:40.445622 434 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:58:40.448958 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:40.449859 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:40.706212 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:40.722151 976 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:40.723436 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:58:40.724052 911 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.724097 2960 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:40.725836 2877 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } }
I20240503 07:58:40.726259 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } }
I20240503 07:58:40.728225 438 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:40.731102 976 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:40.732611 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:40.734344 2960 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:58:40.734715 911 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.735880 2877 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:58:40.736232 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
W20240503 07:58:40.738137 910 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af4caa7f67b9473192241d4030f07f99. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:40.738196 438 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:40.738772 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af4caa7f67b9473192241d4030f07f99 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 07:58:40.938019 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:41.017732 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:41.108229 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:41.247335 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:41.303005 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:41.341713 2980 ts_tablet_manager.cc:902] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:41.342427 2980 tablet_copy_client.cc:308] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:58:41.346261 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet af4caa7f67b9473192241d4030f07f99 from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:55979)
I20240503 07:58:41.346423 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet af4caa7f67b9473192241d4030f07f99 from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:55979: session id = 9e85f3485213464fb71cb81d65aedbad-af4caa7f67b9473192241d4030f07f99
I20240503 07:58:41.347632 996 tablet_copy_source_session.cc:205] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:41.351094 2980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4caa7f67b9473192241d4030f07f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:41.357106 2980 tablet_copy_client.cc:783] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:41.357286 2980 tablet_copy_client.cc:647] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:41.358680 2983 tablet_copy_client.cc:1002] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:41.659416 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:42.600319 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:42.727499 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:42.754133 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:42.796481 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:42.859133 2980 tablet_copy_client.cc:523] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:42.861129 2980 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:58:42.866731 2980 log.cc:824] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:42.886828 2980 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:42.887250 2980 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:58:42.887593 2980 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.027s user 0.028s sys 0.000s
I20240503 07:58:42.889807 2980 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:42.890007 2980 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:58:42.890306 2980 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:42.891151 2980 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:42.891984 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-af4caa7f67b9473192241d4030f07f99 received from {username='slave'} at 127.0.99.133:55979
I20240503 07:58:42.892091 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-af4caa7f67b9473192241d4030f07f99 on tablet af4caa7f67b9473192241d4030f07f99 with peer 9e85f3485213464fb71cb81d65aedbad
W20240503 07:58:42.997568 911 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:43.090062 2012 raft_consensus.cc:1203] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 07:58:43.090255 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:58:43.366806 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:58:43.429699 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:58:43.511643 3052 raft_consensus.cc:1050] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:58:43.512091 3052 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:58:43.513485 2012 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:43.513485 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:58:43.513480 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240503 07:58:43.513903 3054 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:43.514187 3041 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:58:43.514461 911 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:43.516619 3054 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:43.517132 2012 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:43.518469 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
W20240503 07:58:43.518996 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:58:43.519191 432 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 5 to 6, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.523046 976 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:58:43.524020 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:43.524085 2012 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:43.524336 3041 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:43.524830 3041 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:43.525890 3052 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:43.526417 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:43.527129 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af4caa7f67b9473192241d4030f07f99 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:43.527614 2012 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:43.528066 438 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.759181 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 45 similar messages]
I20240503 07:58:43.830907 713 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:58:43.832222 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:58:43.832610 3075 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:58:43.833092 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:43.834259 2929 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } }
I20240503 07:58:43.834686 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } }
I20240503 07:58:43.837105 432 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } }
I20240503 07:58:43.840866 713 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
W20240503 07:58:43.841588 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:58:43.842483 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:43.842718 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:43.843349 3077 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:43.844771 2929 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:43.845932 427 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4c2bc8609e4945148a50dde608502f67 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 07:58:43.846890 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:43.846992 438 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 5 to 6, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:43.848735 648 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213): Couldn't send request to peer 3a4fb386b1bb4b7fa96abbe20f05fbed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c2bc8609e4945148a50dde608502f67. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:43.855175 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 07:58:43.880339 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af4caa7f67b9473192241d4030f07f99: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:43.928615 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:58:44.254789 3096 ts_tablet_manager.cc:902] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Initiating tablet copy from peer 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:44.255671 3096 tablet_copy_client.cc:308] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38831
I20240503 07:58:44.259805 734 tablet_copy_service.cc:140] P 7a36d94f14d04d74b632bc75d1779bc3: Received BeginTabletCopySession request for tablet 4c2bc8609e4945148a50dde608502f67 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed ({username='slave'} at 127.0.99.134:55341)
I20240503 07:58:44.259897 734 tablet_copy_service.cc:161] P 7a36d94f14d04d74b632bc75d1779bc3: Beginning new tablet copy session on tablet 4c2bc8609e4945148a50dde608502f67 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed at {username='slave'} at 127.0.99.134:55341: session id = 3a4fb386b1bb4b7fa96abbe20f05fbed-4c2bc8609e4945148a50dde608502f67
I20240503 07:58:44.260629 734 tablet_copy_source_session.cc:205] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:44.263561 3096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:44.268571 3096 tablet_copy_client.cc:783] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:44.268741 3096 tablet_copy_client.cc:647] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:44.269994 3097 tablet_copy_client.cc:1002] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:45.028620 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:58:45.420413 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:58:45.571362 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:58:45.725617 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 07:58:45.726406 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af4caa7f67b9473192241d4030f07f99: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:45.748970 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:58:45.770416 3096 tablet_copy_client.cc:523] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.772469 3096 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap starting.
I20240503 07:58:45.777706 3096 log.cc:824] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:45.797446 3096 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.797732 3096 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap complete.
I20240503 07:58:45.797987 3096 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.000s
I20240503 07:58:45.800061 3096 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:45.800273 3096 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Initialized, Role: LEARNER
I20240503 07:58:45.800547 3096 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:45.801251 3096 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 07:58:45.801910 734 tablet_copy_service.cc:342] P 7a36d94f14d04d74b632bc75d1779bc3: Request end of tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-4c2bc8609e4945148a50dde608502f67 received from {username='slave'} at 127.0.99.134:55341
I20240503 07:58:45.802000 734 tablet_copy_service.cc:433] P 7a36d94f14d04d74b632bc75d1779bc3: ending tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-4c2bc8609e4945148a50dde608502f67 on tablet 4c2bc8609e4945148a50dde608502f67 with peer 3a4fb386b1bb4b7fa96abbe20f05fbed
W20240503 07:58:45.972111 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:58:46.207398 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:58:46.313482 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:58:46.313711 2143 raft_consensus.cc:1203] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 07:58:46.403945 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:58:46.455837 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:58:46.882372 3127 raft_consensus.cc:1050] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: attempting to promote NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed to VOTER
I20240503 07:58:46.882802 3127 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:58:46.884332 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEARNER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:58:46.884706 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:58:46.884824 3177 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:46.885110 3179 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:58:46.885401 649 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:46.887459 3177 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.888016 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.889302 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.890695 432 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 6 to 7, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.894058 713 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:58:46.895077 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:58:46.895341 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:58:46.895382 3127 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:46.895641 3127 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:46.896751 3177 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.897148 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.897651 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:46.898528 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c2bc8609e4945148a50dde608502f67 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:46.899174 438 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 7 to 8, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
W20240503 07:58:46.902439 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4c2bc8609e4945148a50dde608502f67: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:47.146755 845 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } }
I20240503 07:58:47.148275 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:58:47.148463 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:47.148684 3201 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:47.150322 2932 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:47.150681 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } }
I20240503 07:58:47.152735 432 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:47.155841 845 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:47.157410 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:47.157761 3208 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:58:47.158442 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:47.159489 3201 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:47.159826 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:47.161684 427 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1f0a00c4bec748bc957283814ca26ae1 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 07:58:47.161895 434 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
W20240503 07:58:47.163496 778 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427): Couldn't send request to peer 164b774d3a3346bca3572f022142e17c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f0a00c4bec748bc957283814ca26ae1. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:47.533272 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:58:47.730746 3223 ts_tablet_manager.cc:902] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Initiating tablet copy from peer 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:47.731501 3223 tablet_copy_client.cc:308] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44423
I20240503 07:58:47.735167 865 tablet_copy_service.cc:140] P 5cef4772b54342fe9d418f103c30fc33: Received BeginTabletCopySession request for tablet 1f0a00c4bec748bc957283814ca26ae1 from peer 164b774d3a3346bca3572f022142e17c ({username='slave'} at 127.0.99.136:56809)
I20240503 07:58:47.735303 865 tablet_copy_service.cc:161] P 5cef4772b54342fe9d418f103c30fc33: Beginning new tablet copy session on tablet 1f0a00c4bec748bc957283814ca26ae1 from peer 164b774d3a3346bca3572f022142e17c at {username='slave'} at 127.0.99.136:56809: session id = 164b774d3a3346bca3572f022142e17c-1f0a00c4bec748bc957283814ca26ae1
I20240503 07:58:47.736426 865 tablet_copy_source_session.cc:205] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:47.739697 3223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0a00c4bec748bc957283814ca26ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:47.745074 3223 tablet_copy_client.cc:783] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:47.745257 3223 tablet_copy_client.cc:647] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:47.746667 3225 tablet_copy_client.cc:1002] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:47.759770 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:58:47.833787 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:48.053588 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4c2bc8609e4945148a50dde608502f67: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:48.229820 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:48.560804 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 07:58:48.818614 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:48.933939 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 07:58:48.934504 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:48.952541 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:49.109015 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4c2bc8609e4945148a50dde608502f67: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:49.113885 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240503 07:58:49.247155 3223 tablet_copy_client.cc:523] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:49.249058 3223 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Bootstrap starting.
I20240503 07:58:49.254545 3223 log.cc:824] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:49.275367 3223 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:49.275702 3223 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Bootstrap complete.
I20240503 07:58:49.275987 3223 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.005s
I20240503 07:58:49.277902 3223 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:49.278117 3223 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Initialized, Role: LEARNER
I20240503 07:58:49.278417 3223 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:49.279278 3223 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 07:58:49.279994 865 tablet_copy_service.cc:342] P 5cef4772b54342fe9d418f103c30fc33: Request end of tablet copy session 164b774d3a3346bca3572f022142e17c-1f0a00c4bec748bc957283814ca26ae1 received from {username='slave'} at 127.0.99.136:56809
I20240503 07:58:49.280083 865 tablet_copy_service.cc:433] P 5cef4772b54342fe9d418f103c30fc33: ending tablet copy session 164b774d3a3346bca3572f022142e17c-1f0a00c4bec748bc957283814ca26ae1 on tablet 1f0a00c4bec748bc957283814ca26ae1 with peer 164b774d3a3346bca3572f022142e17c
W20240503 07:58:49.627483 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:49.729885 2405 raft_consensus.cc:1203] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 07:58:49.856379 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:58:50.261128 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:58:50.294507 3240 raft_consensus.cc:1050] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: attempting to promote NON_VOTER 164b774d3a3346bca3572f022142e17c to VOTER
I20240503 07:58:50.294987 3240 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:50.296572 2405 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:50.296872 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 07:58:50.297125 780 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:50.297204 3307 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:50.297343 3240 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:50.299989 3240 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.300516 2405 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.301888 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.302497 434 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:50.306185 845 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:50.307204 2405 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:50.307273 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:50.307560 3307 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:50.307705 3304 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:50.308877 3307 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.309298 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.309530 2405 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:50.310060 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f0a00c4bec748bc957283814ca26ae1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:50.311342 432 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
W20240503 07:58:50.314561 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1f0a00c4bec748bc957283814ca26ae1: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:50.439745 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:58:50.670365 976 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:58:50.671679 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:58:50.672093 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:50.672109 3329 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:50.673707 3128 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } }
I20240503 07:58:50.674161 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } }
I20240503 07:58:50.676066 434 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:50.678898 976 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:50.680509 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:50.681330 3336 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:58:50.681512 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:50.682998 3128 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:50.683372 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:50.684322 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 611c37485cd24fae9374586252925abf with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 07:58:50.685072 434 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:50.685887 909 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427): Couldn't send request to peer 164b774d3a3346bca3572f022142e17c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 611c37485cd24fae9374586252925abf. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:58:50.791615 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:58:50.922859 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 15 similar messages]
I20240503 07:58:51.212092 3349 ts_tablet_manager.cc:902] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:51.212553 3349 tablet_copy_client.cc:308] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:58:51.215992 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 611c37485cd24fae9374586252925abf from peer 164b774d3a3346bca3572f022142e17c ({username='slave'} at 127.0.99.136:33675)
I20240503 07:58:51.216109 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 611c37485cd24fae9374586252925abf from peer 164b774d3a3346bca3572f022142e17c at {username='slave'} at 127.0.99.136:33675: session id = 164b774d3a3346bca3572f022142e17c-611c37485cd24fae9374586252925abf
I20240503 07:58:51.239665 996 tablet_copy_source_session.cc:205] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:51.240625 3349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:51.242537 3349 tablet_copy_client.cc:783] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:51.242727 3349 tablet_copy_client.cc:647] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:51.243781 3353 tablet_copy_client.cc:1002] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:51.445583 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1f0a00c4bec748bc957283814ca26ae1: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:51.465660 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:58:51.519598 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:58:52.230849 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:58:52.488142 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1f0a00c4bec748bc957283814ca26ae1: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:52.672410 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:58:52.729449 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:58:52.744056 3349 tablet_copy_client.cc:523] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:52.745332 3349 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Bootstrap starting.
I20240503 07:58:52.763324 3349 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:52.763590 3349 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Bootstrap complete.
I20240503 07:58:52.763748 3349 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240503 07:58:52.764232 3349 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:52.764343 3349 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Initialized, Role: LEARNER
I20240503 07:58:52.764437 3349 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:52.764737 3349 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:52.765369 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 164b774d3a3346bca3572f022142e17c-611c37485cd24fae9374586252925abf received from {username='slave'} at 127.0.99.136:33675
I20240503 07:58:52.765470 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 164b774d3a3346bca3572f022142e17c-611c37485cd24fae9374586252925abf on tablet 611c37485cd24fae9374586252925abf with peer 164b774d3a3346bca3572f022142e17c
I20240503 07:58:52.766628 2405 raft_consensus.cc:1203] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 07:58:53.056071 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:58:53.207633 3417 raft_consensus.cc:1050] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 164b774d3a3346bca3572f022142e17c to VOTER
I20240503 07:58:53.208127 3417 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:53.209487 2405 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:53.209868 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:53.209864 3419 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:53.210289 3419 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:58:53.210669 911 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:53.212266 3419 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.212990 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.214352 2405 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.214696 438 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 5 to 6, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:53.218225 976 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:53.219231 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:53.219334 2405 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:58:53.219635 3419 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:53.219929 3419 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:58:53.220932 3422 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.221343 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.222101 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 611c37485cd24fae9374586252925abf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:53.222957 2405 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:53.223136 432 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 6 to 7, VOTER 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:58:53.578840 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 611c37485cd24fae9374586252925abf: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:53.583369 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:58:53.617609 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.619339 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.621163 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:53.622994 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:53.623258 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:53.627065 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.649338 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:58:53.689457 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 07:58:53.725800 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.730746 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:58:53.730861 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:53.732031 2123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.733201 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.739794 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:53.751751 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:53.786052 976 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:53.787638 845 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:53.787745 713 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:53.788172 3518 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:53.788552 3422 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:53.790578 3518 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
W20240503 07:58:53.791086 911 consensus_peers.cc:485] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843): Couldn't send request to peer b8f135eb22964d4a8d8300956bce8017. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dac1e203d8a47ae8cbc331ffe5f211b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:53.791214 713 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:53.791661 845 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:53.794035 438 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:53.800806 976 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:53.802158 2012 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:58:53.802186 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:58:53.802601 3417 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:53.803083 3417 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:53.805375 3518 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:53.805785 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
W20240503 07:58:53.806150 910 consensus_peers.cc:485] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213): Couldn't send request to peer 3a4fb386b1bb4b7fa96abbe20f05fbed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af4caa7f67b9473192241d4030f07f99. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:53.806499 2012 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:53.807981 432 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:53.815989 713 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:53.817409 976 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:58:53.817570 2274 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:58:53.818046 3544 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:53.818255 3545 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:53.820701 3545 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
W20240503 07:58:53.821066 648 consensus_peers.cc:485] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eca5e6693b24d0e8f3d9c37a62a1bca. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:53.821144 2274 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:58:53.821286 976 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:58:53.823289 438 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 7 to 8, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
W20240503 07:58:53.847051 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:58:53.876992 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:58:54.132900 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 07:58:54.186511 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240503 07:58:54.212533 3579 ts_tablet_manager.cc:902] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:54.213016 3579 tablet_copy_client.cc:308] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:58:54.216300 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 2dac1e203d8a47ae8cbc331ffe5f211b from peer b8f135eb22964d4a8d8300956bce8017 ({username='slave'} at 127.0.99.135:32837)
I20240503 07:58:54.216393 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 2dac1e203d8a47ae8cbc331ffe5f211b from peer b8f135eb22964d4a8d8300956bce8017 at {username='slave'} at 127.0.99.135:32837: session id = b8f135eb22964d4a8d8300956bce8017-2dac1e203d8a47ae8cbc331ffe5f211b
I20240503 07:58:54.217145 996 tablet_copy_source_session.cc:205] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:54.217959 3579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dac1e203d8a47ae8cbc331ffe5f211b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:54.219854 3579 tablet_copy_client.cc:783] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:54.219998 3579 tablet_copy_client.cc:647] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:54.221012 3583 tablet_copy_client.cc:1002] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: tablet copy: Injecting latency into file download: 1500
I20240503 07:58:54.226889 3585 ts_tablet_manager.cc:902] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:54.227404 3585 tablet_copy_client.cc:308] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38831
I20240503 07:58:54.230778 734 tablet_copy_service.cc:140] P 7a36d94f14d04d74b632bc75d1779bc3: Received BeginTabletCopySession request for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:53237)
I20240503 07:58:54.230868 734 tablet_copy_service.cc:161] P 7a36d94f14d04d74b632bc75d1779bc3: Beginning new tablet copy session on tablet 1eca5e6693b24d0e8f3d9c37a62a1bca from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:53237: session id = 9e85f3485213464fb71cb81d65aedbad-1eca5e6693b24d0e8f3d9c37a62a1bca
I20240503 07:58:54.231557 734 tablet_copy_source_session.cc:205] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:54.232275 3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:54.233893 3585 tablet_copy_client.cc:783] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:54.234028 3585 tablet_copy_client.cc:647] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:54.235049 3589 tablet_copy_client.cc:1002] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
I20240503 07:58:54.305733 3606 ts_tablet_manager.cc:902] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:54.306203 3606 tablet_copy_client.cc:308] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:58:54.309500 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet af4caa7f67b9473192241d4030f07f99 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed ({username='slave'} at 127.0.99.134:56821)
I20240503 07:58:54.309590 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet af4caa7f67b9473192241d4030f07f99 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed at {username='slave'} at 127.0.99.134:56821: session id = 3a4fb386b1bb4b7fa96abbe20f05fbed-af4caa7f67b9473192241d4030f07f99
I20240503 07:58:54.310268 996 tablet_copy_source_session.cc:205] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:54.311008 3606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4caa7f67b9473192241d4030f07f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:54.312744 3606 tablet_copy_client.cc:783] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:54.312889 3606 tablet_copy_client.cc:647] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:54.313861 3610 tablet_copy_client.cc:1002] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:54.788157 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:58:55.123077 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:58:55.234843 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:58:55.300017 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4c2bc8609e4945148a50dde608502f67: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:58:55.721304 3579 tablet_copy_client.cc:523] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:55.722635 3579 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Bootstrap starting.
I20240503 07:58:55.735500 3585 tablet_copy_client.cc:523] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:55.737071 3585 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:58:55.741725 3579 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:55.741971 3579 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Bootstrap complete.
I20240503 07:58:55.742121 3579 ts_tablet_manager.cc:1367] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20240503 07:58:55.742631 3579 raft_consensus.cc:348] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:55.742754 3579 raft_consensus.cc:729] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f135eb22964d4a8d8300956bce8017, State: Initialized, Role: LEARNER
I20240503 07:58:55.742873 3579 consensus_queue.cc:260] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:55.743218 3579 ts_tablet_manager.cc:1398] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:55.744180 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session b8f135eb22964d4a8d8300956bce8017-2dac1e203d8a47ae8cbc331ffe5f211b received from {username='slave'} at 127.0.99.135:32837
I20240503 07:58:55.744256 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session b8f135eb22964d4a8d8300956bce8017-2dac1e203d8a47ae8cbc331ffe5f211b on tablet 2dac1e203d8a47ae8cbc331ffe5f211b with peer b8f135eb22964d4a8d8300956bce8017
I20240503 07:58:55.761502 3585 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:55.761781 3585 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:58:55.761936 3585 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.001s
I20240503 07:58:55.762598 3585 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:55.762717 3585 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:58:55.762805 3585 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:58:55.763098 3585 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.001s user 0.001s sys 0.003s
I20240503 07:58:55.763681 734 tablet_copy_service.cc:342] P 7a36d94f14d04d74b632bc75d1779bc3: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-1eca5e6693b24d0e8f3d9c37a62a1bca received from {username='slave'} at 127.0.99.133:53237
I20240503 07:58:55.763779 734 tablet_copy_service.cc:433] P 7a36d94f14d04d74b632bc75d1779bc3: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-1eca5e6693b24d0e8f3d9c37a62a1bca on tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with peer 9e85f3485213464fb71cb81d65aedbad
I20240503 07:58:55.814194 3606 tablet_copy_client.cc:523] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:55.815671 3606 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap starting.
I20240503 07:58:55.833746 3606 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:55.833962 3606 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap complete.
I20240503 07:58:55.834098 3606 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.002s
I20240503 07:58:55.834553 3606 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:55.834666 3606 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Initialized, Role: LEARNER
I20240503 07:58:55.834743 3606 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:55.835073 3606 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:58:55.835597 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-af4caa7f67b9473192241d4030f07f99 received from {username='slave'} at 127.0.99.134:56821
I20240503 07:58:55.835673 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-af4caa7f67b9473192241d4030f07f99 on tablet af4caa7f67b9473192241d4030f07f99 with peer 3a4fb386b1bb4b7fa96abbe20f05fbed
I20240503 07:58:55.848043 2274 raft_consensus.cc:1203] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:55.944720 2012 raft_consensus.cc:1203] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 07:58:55.988560 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:58:56.239748 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:58:56.240199 2143 raft_consensus.cc:1203] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 07:58:56.321467 3772 raft_consensus.cc:1050] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:58:56.321954 3772 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:58:56.323284 2012 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.323284 975 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.323330 2274 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.323679 3774 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.323959 3771 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.324127 3772 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.326143 3649 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.326493 2274 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.326510 2012 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.328178 975 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.328961 432 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad reported cstate change: config changed from index 8 to 9, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.332929 713 consensus_queue.cc:237] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:58:56.333948 2274 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:58:56.334048 2012 raft_consensus.cc:1261] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:58:56.334383 3772 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:56.334553 3771 consensus_queue.cc:1035] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:56.335979 3774 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.336380 2274 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.336381 2012 raft_consensus.cc:2932] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.337880 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:56.338456 434 catalog_manager.cc:5484] T 1eca5e6693b24d0e8f3d9c37a62a1bca P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 9 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:58:56.340931 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca with delete_type TABLET_DATA_TOMBSTONED (TS 80bd2b9fafaf44bcb8c0f1b9f408841e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43524
I20240503 07:58:56.341979 3782 tablet_replica.cc:330] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: stopping tablet replica
I20240503 07:58:56.342278 3782 raft_consensus.cc:2229] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:56.342495 3782 raft_consensus.cc:2258] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:56.343667 3782 ts_tablet_manager.cc:1875] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:56.345464 3782 ts_tablet_manager.cc:1888] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 07:58:56.345538 3782 log.cc:1196] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1eca5e6693b24d0e8f3d9c37a62a1bca
I20240503 07:58:56.346045 428 catalog_manager.cc:4830] TS 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363): tablet 1eca5e6693b24d0e8f3d9c37a62a1bca (table rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]) successfully deleted
I20240503 07:58:56.363507 3783 raft_consensus.cc:1050] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER b8f135eb22964d4a8d8300956bce8017 to VOTER
I20240503 07:58:56.363987 3783 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:56.365447 2274 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:56.365573 713 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:56.365597 845 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:56.365900 3770 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:56.366124 3785 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:56.366317 3783 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:56.368198 3785 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.368638 2274 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.369055 845 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.371042 438 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:56.371734 713 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.374220 975 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:56.375445 2274 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:56.375536 845 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:56.375820 3612 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:56.375991 3770 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:56.377388 3783 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.377820 845 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.377823 2274 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.379014 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dac1e203d8a47ae8cbc331ffe5f211b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:56.379819 435 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:56.381898 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dac1e203d8a47ae8cbc331ffe5f211b with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38956
I20240503 07:58:56.382441 3789 tablet_replica.cc:330] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:58:56.382767 3789 raft_consensus.cc:2229] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:56.383054 3789 raft_consensus.cc:2258] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:56.384188 3789 ts_tablet_manager.cc:1875] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:56.385986 3789 ts_tablet_manager.cc:1888] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:58:56.386056 3789 log.cc:1196] T 2dac1e203d8a47ae8cbc331ffe5f211b P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2dac1e203d8a47ae8cbc331ffe5f211b
I20240503 07:58:56.386559 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 2dac1e203d8a47ae8cbc331ffe5f211b (table rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]) successfully deleted
W20240503 07:58:56.389976 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:58:56.455219 713 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:56.456763 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:58:56.456920 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:58:56.457190 3649 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.457733 3649 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.459425 3772 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:56.460337 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
W20240503 07:58:56.460664 647 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427): Couldn't send request to peer 164b774d3a3346bca3572f022142e17c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c2bc8609e4945148a50dde608502f67. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:56.460840 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:58:56.463182 438 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 8 to 9, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:56.469975 845 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:56.471542 975 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:56.471802 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:58:56.471905 3304 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:56.472249 3821 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:58:56.473533 3304 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:56.474416 713 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:56.475102 975 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
W20240503 07:58:56.475889 780 consensus_peers.cc:485] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 -> Peer b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843): Couldn't send request to peer b8f135eb22964d4a8d8300956bce8017. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71d661e6c1374f9282f8eff323861e1e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:56.477100 434 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index -1 to 4, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:58:56.756033 3612 raft_consensus.cc:1050] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed to VOTER
I20240503 07:58:56.756510 3612 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:58:56.757984 2143 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.758025 2012 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.758034 845 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:58:56.758424 3612 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.758616 3783 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.758901 3785 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:58:56.760670 3785 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.761062 2143 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.762847 845 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.762845 2012 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.763340 432 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.766557 975 consensus_queue.cc:237] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:58:56.767589 2143 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:58:56.767629 2012 raft_consensus.cc:1261] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:58:56.767953 3612 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:56.768190 3785 consensus_queue.cc:1035] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:56.769713 3770 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.770131 2143 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.771210 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af4caa7f67b9473192241d4030f07f99 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:56.771781 2012 raft_consensus.cc:2932] T af4caa7f67b9473192241d4030f07f99 P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:58:56.772024 434 catalog_manager.cc:5484] T af4caa7f67b9473192241d4030f07f99 P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 9 to 10, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:56.774633 812 tablet_service.cc:1514] Processing DeleteTablet for tablet af4caa7f67b9473192241d4030f07f99 with delete_type TABLET_DATA_TOMBSTONED (TS 5cef4772b54342fe9d418f103c30fc33 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34178
I20240503 07:58:56.775364 3860 tablet_replica.cc:330] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: stopping tablet replica
I20240503 07:58:56.775660 3860 raft_consensus.cc:2229] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:56.775882 3860 raft_consensus.cc:2258] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:56.777097 3860 ts_tablet_manager.cc:1875] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:56.778995 3860 ts_tablet_manager.cc:1888] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 07:58:56.779076 3860 log.cc:1196] T af4caa7f67b9473192241d4030f07f99 P 5cef4772b54342fe9d418f103c30fc33: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/af4caa7f67b9473192241d4030f07f99
I20240503 07:58:56.779561 427 catalog_manager.cc:4830] TS 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423): tablet af4caa7f67b9473192241d4030f07f99 (table rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]) successfully deleted
I20240503 07:58:56.892596 3861 ts_tablet_manager.cc:902] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Initiating tablet copy from peer 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:58:56.893114 3861 tablet_copy_client.cc:308] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44423
I20240503 07:58:56.896420 865 tablet_copy_service.cc:140] P 5cef4772b54342fe9d418f103c30fc33: Received BeginTabletCopySession request for tablet 71d661e6c1374f9282f8eff323861e1e from peer b8f135eb22964d4a8d8300956bce8017 ({username='slave'} at 127.0.99.135:48611)
I20240503 07:58:56.896539 865 tablet_copy_service.cc:161] P 5cef4772b54342fe9d418f103c30fc33: Beginning new tablet copy session on tablet 71d661e6c1374f9282f8eff323861e1e from peer b8f135eb22964d4a8d8300956bce8017 at {username='slave'} at 127.0.99.135:48611: session id = b8f135eb22964d4a8d8300956bce8017-71d661e6c1374f9282f8eff323861e1e
I20240503 07:58:56.897274 865 tablet_copy_source_session.cc:205] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:56.898387 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d661e6c1374f9282f8eff323861e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:56.900499 3861 tablet_copy_client.cc:783] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:56.900676 3861 tablet_copy_client.cc:647] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:56.901719 3865 tablet_copy_client.cc:1002] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Injecting latency into file download: 1500
I20240503 07:58:57.078872 3892 ts_tablet_manager.cc:902] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Initiating tablet copy from peer 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831)
I20240503 07:58:57.079365 3892 tablet_copy_client.cc:308] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38831
I20240503 07:58:57.083288 734 tablet_copy_service.cc:140] P 7a36d94f14d04d74b632bc75d1779bc3: Received BeginTabletCopySession request for tablet 4c2bc8609e4945148a50dde608502f67 from peer 164b774d3a3346bca3572f022142e17c ({username='slave'} at 127.0.99.136:37045)
I20240503 07:58:57.083417 734 tablet_copy_service.cc:161] P 7a36d94f14d04d74b632bc75d1779bc3: Beginning new tablet copy session on tablet 4c2bc8609e4945148a50dde608502f67 from peer 164b774d3a3346bca3572f022142e17c at {username='slave'} at 127.0.99.136:37045: session id = 164b774d3a3346bca3572f022142e17c-4c2bc8609e4945148a50dde608502f67
I20240503 07:58:57.084273 734 tablet_copy_source_session.cc:205] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:57.085100 3892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:57.086874 3892 tablet_copy_client.cc:783] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:57.087172 3892 tablet_copy_client.cc:647] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:57.088204 3896 tablet_copy_client.cc:1002] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: tablet copy: Injecting latency into file download: 1500
W20240503 07:58:57.169127 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240503 07:58:57.169865 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:57.270463 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:58:57.605541 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:58:57.897715 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:58:58.268798 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 07:58:58.402024 3861 tablet_copy_client.cc:523] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:58.403510 3861 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Bootstrap starting.
W20240503 07:58:58.412078 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 07:58:58.422362 3861 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:58.422645 3861 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Bootstrap complete.
I20240503 07:58:58.422834 3861 ts_tablet_manager.cc:1367] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.006s
I20240503 07:58:58.423364 3861 raft_consensus.cc:348] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:58.423524 3861 raft_consensus.cc:729] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f135eb22964d4a8d8300956bce8017, State: Initialized, Role: LEARNER
I20240503 07:58:58.423643 3861 consensus_queue.cc:260] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:58:58.423974 3861 ts_tablet_manager.cc:1398] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:58:58.424499 865 tablet_copy_service.cc:342] P 5cef4772b54342fe9d418f103c30fc33: Request end of tablet copy session b8f135eb22964d4a8d8300956bce8017-71d661e6c1374f9282f8eff323861e1e received from {username='slave'} at 127.0.99.135:48611
I20240503 07:58:58.424582 865 tablet_copy_service.cc:433] P 5cef4772b54342fe9d418f103c30fc33: ending tablet copy session b8f135eb22964d4a8d8300956bce8017-71d661e6c1374f9282f8eff323861e1e on tablet 71d661e6c1374f9282f8eff323861e1e with peer b8f135eb22964d4a8d8300956bce8017
I20240503 07:58:58.588529 3892 tablet_copy_client.cc:523] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:58.589880 3892 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Bootstrap starting.
I20240503 07:58:58.608572 3892 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:58.608845 3892 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Bootstrap complete.
I20240503 07:58:58.609005 3892 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.003s
I20240503 07:58:58.609401 3892 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:58.609510 3892 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Initialized, Role: LEARNER
I20240503 07:58:58.609613 3892 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:58:58.609930 3892 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:58:58.610440 734 tablet_copy_service.cc:342] P 7a36d94f14d04d74b632bc75d1779bc3: Request end of tablet copy session 164b774d3a3346bca3572f022142e17c-4c2bc8609e4945148a50dde608502f67 received from {username='slave'} at 127.0.99.136:37045
I20240503 07:58:58.610517 734 tablet_copy_service.cc:433] P 7a36d94f14d04d74b632bc75d1779bc3: ending tablet copy session 164b774d3a3346bca3572f022142e17c-4c2bc8609e4945148a50dde608502f67 on tablet 4c2bc8609e4945148a50dde608502f67 with peer 164b774d3a3346bca3572f022142e17c
I20240503 07:58:58.677822 2405 raft_consensus.cc:1203] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240503 07:58:58.679419 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1f0a00c4bec748bc957283814ca26ae1: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:58:58.703315 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 07:58:58.901883 2274 raft_consensus.cc:1203] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 07:58:58.932762 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 07:58:59.081226 3934 raft_consensus.cc:1050] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: attempting to promote NON_VOTER 164b774d3a3346bca3572f022142e17c to VOTER
I20240503 07:58:59.081696 3934 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:59.083153 2405 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 LEARNER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:58:59.083206 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:58:59.083206 845 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:58:59.083546 4018 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:59.083786 4015 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:59.083976 4016 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:58:59.085738 4018 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:59.086135 845 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:59.086190 2405 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:58:59.088253 438 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 9 to 10, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:59.088511 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
W20240503 07:58:59.261098 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 07:58:59.275856 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35134
I20240503 07:58:59.276042 713 raft_consensus.cc:595] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Received request to transfer leadership
I20240503 07:58:59.436707 4043 raft_consensus.cc:1050] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: attempting to promote NON_VOTER b8f135eb22964d4a8d8300956bce8017 to VOTER
I20240503 07:58:59.437212 4043 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:59.438709 2274 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:59.438745 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:59.439122 975 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:59.439234 3923 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:59.439456 4043 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:59.439744 4042 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:59.442164 3923 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.442654 713 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.442795 975 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.445233 432 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.445216 2274 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.448781 845 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:58:59.449855 2274 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:59.449862 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:58:59.450184 4042 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:59.450449 4043 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:59.452028 3923 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.452491 713 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.453410 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71d661e6c1374f9282f8eff323861e1e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:58:59.453727 2274 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.455107 434 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 5 to 6, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:58:59.457669 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d661e6c1374f9282f8eff323861e1e with delete_type TABLET_DATA_TOMBSTONED (TS 80bd2b9fafaf44bcb8c0f1b9f408841e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43524
I20240503 07:58:59.457988 4049 tablet_replica.cc:330] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: stopping tablet replica
I20240503 07:58:59.458120 4049 raft_consensus.cc:2229] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:59.458302 4049 raft_consensus.cc:2258] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:59.458864 4049 ts_tablet_manager.cc:1875] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:59.460515 4049 ts_tablet_manager.cc:1888] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:58:59.460577 4049 log.cc:1196] T 71d661e6c1374f9282f8eff323861e1e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/71d661e6c1374f9282f8eff323861e1e
I20240503 07:58:59.461092 428 catalog_manager.cc:4830] TS 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363): tablet 71d661e6c1374f9282f8eff323861e1e (table rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]) successfully deleted
I20240503 07:58:59.491827 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35158
I20240503 07:58:59.491961 713 raft_consensus.cc:595] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Received request to transfer leadership
I20240503 07:58:59.499491 975 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:58:59.500675 4016 raft_consensus.cc:979] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: : Instructing follower 5cef4772b54342fe9d418f103c30fc33 to start an election
I20240503 07:58:59.501003 4016 raft_consensus.cc:1067] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Signalling peer 5cef4772b54342fe9d418f103c30fc33 to start an election
I20240503 07:58:59.501139 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:58:59.501463 2405 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 07:58:59.501761 910 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213): Couldn't send request to peer 3a4fb386b1bb4b7fa96abbe20f05fbed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 611c37485cd24fae9374586252925abf. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:59.501868 4073 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:59.502200 4073 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:58:59.502323 845 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
from {username='slave'} at 127.0.99.130:47687
I20240503 07:58:59.502475 845 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:59.502547 845 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:58:59.503563 845 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:59.504065 845 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 3 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213), 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427)
I20240503 07:58:59.504122 4072 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:59.504616 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "164b774d3a3346bca3572f022142e17c"
I20240503 07:58:59.504788 2405 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:58:59.505651 714 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:59.505834 2405 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 3.
I20240503 07:58:59.507115 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:58:59.507190 438 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.507341 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:58:59.507438 713 raft_consensus.cc:3032] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:58:59.507475 713 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:58:59.507622 713 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:59.507791 713 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:58:59.508616 713 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 3.
I20240503 07:58:59.508924 780 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 164b774d3a3346bca3572f022142e17c, 5cef4772b54342fe9d418f103c30fc33, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:58:59.509235 4042 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:58:59.509330 4042 raft_consensus.cc:686] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Becoming Leader. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Running, Role: LEADER
I20240503 07:58:59.509459 4042 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:58:59.511343 432 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: term changed from 2 to 3, leader changed from 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) to 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.513321 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "5cef4772b54342fe9d418f103c30fc33" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
I20240503 07:58:59.513494 2143 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:58:59.514453 2143 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cef4772b54342fe9d418f103c30fc33 in term 3.
W20240503 07:58:59.609216 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 07:58:59.727226 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 07:58:59.929159 4099 ts_tablet_manager.cc:902] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:58:59.929663 4099 tablet_copy_client.cc:308] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:58:59.930159 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 611c37485cd24fae9374586252925abf from peer 3a4fb386b1bb4b7fa96abbe20f05fbed ({username='slave'} at 127.0.99.134:56821)
I20240503 07:58:59.930277 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 611c37485cd24fae9374586252925abf from peer 3a4fb386b1bb4b7fa96abbe20f05fbed at {username='slave'} at 127.0.99.134:56821: session id = 3a4fb386b1bb4b7fa96abbe20f05fbed-611c37485cd24fae9374586252925abf
I20240503 07:58:59.931172 996 tablet_copy_source_session.cc:205] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:59.932013 4099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.933704 4099 tablet_copy_client.cc:783] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:59.933861 4099 tablet_copy_client.cc:647] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:59.934996 4103 tablet_copy_client.cc:1002] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Injecting latency into file download: 1500
I20240503 07:58:59.972152 2404 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 07:58:59.972573 4043 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:58:59.993355 713 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 07:58:59.993882 4043 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:58:59.996544 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 07:58:59.997198 4043 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 07:59:00.169641 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af4caa7f67b9473192241d4030f07f99: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
W20240503 07:59:00.224191 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 07:59:00.561998 845 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:59:00.563148 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 07:59:00.563216 2404 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 07:59:00.563539 4043 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:00.563724 4148 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:00.565388 4149 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:00.565699 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:00.565747 2404 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:00.567157 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c2bc8609e4945148a50dde608502f67 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:00.568197 435 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 10 to 12, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:59:00.568675 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 07:59:00.570987 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2bc8609e4945148a50dde608502f67 with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:38956
I20240503 07:59:00.571449 4157 tablet_replica.cc:330] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:59:00.571732 4157 raft_consensus.cc:2229] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:00.571854 4157 raft_consensus.cc:2258] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:00.572548 4157 ts_tablet_manager.cc:1875] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:00.574761 4157 ts_tablet_manager.cc:1888] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 07:59:00.574836 4157 log.cc:1196] T 4c2bc8609e4945148a50dde608502f67 P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4c2bc8609e4945148a50dde608502f67
I20240503 07:59:00.575368 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 4c2bc8609e4945148a50dde608502f67 (table rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]) successfully deleted
I20240503 07:59:00.587869 975 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:00.589248 845 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:00.589439 713 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:00.589628 4136 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:00.589797 3770 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:00.589969 910 consensus_peers.cc:485] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33a8cbbafd86402a9a3bf9e730b60c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:00.591565 4137 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:00.591926 713 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:00.592240 845 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:00.594312 432 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index -1 to 4, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:00.602829 975 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:00.604161 845 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:00.604157 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:00.604352 909 consensus_peers.cc:485] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427): Couldn't send request to peer 164b774d3a3346bca3572f022142e17c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1f5e5ffc9946639c4da0129d6d479e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:00.604558 3770 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:00.604744 4176 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:00.606297 4136 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:00.606727 713 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:00.606781 845 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:00.608721 435 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index -1 to 4, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:00.696547 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 07:59:00.930481 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 07:59:00.985242 4215 ts_tablet_manager.cc:902] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:59:00.985752 4215 tablet_copy_client.cc:308] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:59:00.986260 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet ae1f5e5ffc9946639c4da0129d6d479e from peer 164b774d3a3346bca3572f022142e17c ({username='slave'} at 127.0.99.136:33675)
I20240503 07:59:00.986377 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet ae1f5e5ffc9946639c4da0129d6d479e from peer 164b774d3a3346bca3572f022142e17c at {username='slave'} at 127.0.99.136:33675: session id = 164b774d3a3346bca3572f022142e17c-ae1f5e5ffc9946639c4da0129d6d479e
I20240503 07:59:00.987188 996 tablet_copy_source_session.cc:205] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:00.988085 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f5e5ffc9946639c4da0129d6d479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.989780 4215 tablet_copy_client.cc:783] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:00.989925 4215 tablet_copy_client.cc:647] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:00.991035 4219 tablet_copy_client.cc:1002] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:01.210299 4235 ts_tablet_manager.cc:902] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:59:01.210736 4235 tablet_copy_client.cc:308] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:59:01.211283 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 33a8cbbafd86402a9a3bf9e730b60c0e from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:55979)
I20240503 07:59:01.211381 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 33a8cbbafd86402a9a3bf9e730b60c0e from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:55979: session id = 9e85f3485213464fb71cb81d65aedbad-33a8cbbafd86402a9a3bf9e730b60c0e
I20240503 07:59:01.212132 996 tablet_copy_source_session.cc:205] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:01.212944 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a8cbbafd86402a9a3bf9e730b60c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.214671 4235 tablet_copy_client.cc:783] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:01.214818 4235 tablet_copy_client.cc:647] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:01.215809 4238 tablet_copy_client.cc:1002] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:01.435374 4099 tablet_copy_client.cc:523] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:01.436765 4099 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap starting.
I20240503 07:59:01.454737 4099 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:01.455026 4099 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap complete.
I20240503 07:59:01.455205 4099 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240503 07:59:01.455639 4099 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:01.455744 4099 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Initialized, Role: LEARNER
I20240503 07:59:01.455849 4099 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:01.456130 4099 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:01.456693 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-611c37485cd24fae9374586252925abf received from {username='slave'} at 127.0.99.134:56821
I20240503 07:59:01.456792 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-611c37485cd24fae9374586252925abf on tablet 611c37485cd24fae9374586252925abf with peer 3a4fb386b1bb4b7fa96abbe20f05fbed
W20240503 07:59:01.496448 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 07:59:01.642841 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 611c37485cd24fae9374586252925abf: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:01.866860 2143 raft_consensus.cc:1203] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 07:59:02.211514 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 07:59:02.303696 4275 raft_consensus.cc:1050] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed to VOTER
I20240503 07:59:02.304179 4275 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:02.306133 2143 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:02.306162 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:02.307210 4274 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:02.307315 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:02.307590 4274 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:02.308852 4324 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:02.310253 4324 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:02.310703 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:02.312340 2143 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:02.313107 434 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:02.313922 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:02.367375 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "611c37485cd24fae9374586252925abf"
dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42090
I20240503 07:59:02.367563 975 raft_consensus.cc:595] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:02.450029 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 07:59:02.491324 4215 tablet_copy_client.cc:523] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:02.492599 4215 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Bootstrap starting.
I20240503 07:59:02.509644 4215 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:02.509891 4215 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Bootstrap complete.
I20240503 07:59:02.510035 4215 ts_tablet_manager.cc:1367] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 07:59:02.510391 4215 raft_consensus.cc:348] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:02.510496 4215 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Initialized, Role: LEARNER
I20240503 07:59:02.510574 4215 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:02.510805 4215 ts_tablet_manager.cc:1398] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.511407 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 164b774d3a3346bca3572f022142e17c-ae1f5e5ffc9946639c4da0129d6d479e received from {username='slave'} at 127.0.99.136:33675
I20240503 07:59:02.511507 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 164b774d3a3346bca3572f022142e17c-ae1f5e5ffc9946639c4da0129d6d479e on tablet ae1f5e5ffc9946639c4da0129d6d479e with peer 164b774d3a3346bca3572f022142e17c
I20240503 07:59:02.513715 2404 raft_consensus.cc:1203] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:02.589871 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "611c37485cd24fae9374586252925abf"
dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42126
I20240503 07:59:02.590008 975 raft_consensus.cc:595] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:02.716156 4235 tablet_copy_client.cc:523] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:02.717434 4235 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:59:02.734663 4235 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:02.734889 4235 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:59:02.735060 4235 ts_tablet_manager.cc:1367] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240503 07:59:02.735437 4235 raft_consensus.cc:348] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:02.735540 4235 raft_consensus.cc:729] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:59:02.735610 4235 consensus_queue.cc:260] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:02.735949 4235 ts_tablet_manager.cc:1398] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:02.736490 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-33a8cbbafd86402a9a3bf9e730b60c0e received from {username='slave'} at 127.0.99.133:55979
I20240503 07:59:02.736564 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-33a8cbbafd86402a9a3bf9e730b60c0e on tablet 33a8cbbafd86402a9a3bf9e730b60c0e with peer 9e85f3485213464fb71cb81d65aedbad
I20240503 07:59:02.792670 4275 raft_consensus.cc:979] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: : Instructing follower 3a4fb386b1bb4b7fa96abbe20f05fbed to start an election
I20240503 07:59:02.792785 4275 raft_consensus.cc:1067] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Signalling peer 3a4fb386b1bb4b7fa96abbe20f05fbed to start an election
I20240503 07:59:02.793378 2143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "611c37485cd24fae9374586252925abf"
dest_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
from {username='slave'} at 127.0.99.132:58007
I20240503 07:59:02.793576 2143 raft_consensus.cc:482] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:02.793650 2143 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:02.794533 2143 raft_consensus.cc:504] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:02.795610 2143 leader_election.cc:288] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [CANDIDATE]: Term 2 election: Requested vote from peers 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831), 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427)
I20240503 07:59:02.795874 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
I20240503 07:59:02.795954 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:59:02.796002 713 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:02.796070 975 raft_consensus.cc:3032] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:02.796110 975 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:59:02.796257 975 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:02.796427 975 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:02.796965 713 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a4fb386b1bb4b7fa96abbe20f05fbed in term 2.
I20240503 07:59:02.797420 975 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a4fb386b1bb4b7fa96abbe20f05fbed in term 2.
I20240503 07:59:02.797784 2079 leader_election.cc:302] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3a4fb386b1bb4b7fa96abbe20f05fbed, 7a36d94f14d04d74b632bc75d1779bc3, 80bd2b9fafaf44bcb8c0f1b9f408841e; no voters:
I20240503 07:59:02.798023 4258 raft_consensus.cc:2781] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:02.798192 4258 raft_consensus.cc:686] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEADER]: Becoming Leader. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Running, Role: LEADER
I20240503 07:59:02.798492 4258 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:02.800766 2404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611c37485cd24fae9374586252925abf" candidate_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "164b774d3a3346bca3572f022142e17c"
I20240503 07:59:02.800900 2404 raft_consensus.cc:3037] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:02.801494 435 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: term changed from 1 to 2, leader changed from 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) to 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134). New cstate: current_term: 2 leader_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:02.801875 2404 raft_consensus.cc:2454] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a4fb386b1bb4b7fa96abbe20f05fbed in term 2.
I20240503 07:59:02.906184 4275 raft_consensus.cc:1050] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 164b774d3a3346bca3572f022142e17c to VOTER
I20240503 07:59:02.906606 4275 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:59:02.907837 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:02.907955 845 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:02.908033 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:02.908319 4324 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:02.908470 4397 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:02.908641 4395 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:02.910590 4395 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.911041 713 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.911325 2404 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.912739 845 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.913502 435 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 4 to 5, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.916500 975 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:59:02.917522 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:02.917585 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:02.917910 4395 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:02.918087 4324 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:02.919543 4395 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.919905 2404 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.921140 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae1f5e5ffc9946639c4da0129d6d479e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:02.921177 713 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:02.921804 438 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 5 to 6, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:02.924156 812 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1f5e5ffc9946639c4da0129d6d479e with delete_type TABLET_DATA_TOMBSTONED (TS 5cef4772b54342fe9d418f103c30fc33 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34178
I20240503 07:59:02.924650 4401 tablet_replica.cc:330] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: stopping tablet replica
I20240503 07:59:02.924799 4401 raft_consensus.cc:2229] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:02.924966 4401 raft_consensus.cc:2258] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:02.925598 4401 ts_tablet_manager.cc:1875] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:02.927222 4401 ts_tablet_manager.cc:1888] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:02.927280 4401 log.cc:1196] T ae1f5e5ffc9946639c4da0129d6d479e P 5cef4772b54342fe9d418f103c30fc33: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae1f5e5ffc9946639c4da0129d6d479e
I20240503 07:59:02.927671 427 catalog_manager.cc:4830] TS 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423): tablet ae1f5e5ffc9946639c4da0129d6d479e (table rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]) successfully deleted
I20240503 07:59:03.031731 2012 raft_consensus.cc:1203] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 07:59:03.065451 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 07:59:03.142108 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 07:59:03.242033 975 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:03.242542 4258 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:03.392661 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:03.393126 4258 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:03.396181 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:03.397526 4438 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 07:59:03.541616 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 07:59:03.542356 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4c2bc8609e4945148a50dde608502f67: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:03.573743 4460 raft_consensus.cc:1050] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:59:03.574239 4460 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:03.575632 845 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:03.575632 2012 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:03.575693 713 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:03.576002 4459 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:03.576180 4397 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:03.576507 4462 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:03.578687 4459 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.579380 2012 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.579387 845 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.581326 432 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 4 to 5, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:03.581925 713 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.584755 975 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:03.585942 2012 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:03.586012 845 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:03.586385 4460 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:03.586632 4397 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:03.587797 4460 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.588186 845 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.588382 2012 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.589318 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33a8cbbafd86402a9a3bf9e730b60c0e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:03.590230 435 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 5 to 6, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:03.592330 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a8cbbafd86402a9a3bf9e730b60c0e with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38956
I20240503 07:59:03.592711 4470 tablet_replica.cc:330] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:59:03.592862 4470 raft_consensus.cc:2229] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:03.593159 4470 raft_consensus.cc:2258] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:03.593763 4470 ts_tablet_manager.cc:1875] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:03.595369 4470 ts_tablet_manager.cc:1888] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:03.595429 4470 log.cc:1196] T 33a8cbbafd86402a9a3bf9e730b60c0e P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/33a8cbbafd86402a9a3bf9e730b60c0e
I20240503 07:59:03.595830 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 33a8cbbafd86402a9a3bf9e730b60c0e (table rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]) successfully deleted
W20240503 07:59:03.626506 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 07:59:03.713155 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 07:59:03.917502 2143 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:03.918707 713 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 07:59:03.918756 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 07:59:03.919100 4491 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:03.919330 4438 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:03.921034 4492 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:03.921203 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:03.921373 713 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:03.922514 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 611c37485cd24fae9374586252925abf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:03.923846 434 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c reported cstate change: config changed from index 9 to 11, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:03.926445 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 611c37485cd24fae9374586252925abf with delete_type TABLET_DATA_TOMBSTONED (TS 80bd2b9fafaf44bcb8c0f1b9f408841e not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43524
I20240503 07:59:03.926945 4495 tablet_replica.cc:330] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: stopping tablet replica
I20240503 07:59:03.927119 4495 raft_consensus.cc:2229] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:03.927242 4495 raft_consensus.cc:2258] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:03.927815 4495 ts_tablet_manager.cc:1875] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:03.929584 4495 ts_tablet_manager.cc:1888] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 07:59:03.929641 4495 log.cc:1196] T 611c37485cd24fae9374586252925abf P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/611c37485cd24fae9374586252925abf
I20240503 07:59:03.930079 428 catalog_manager.cc:4830] TS 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363): tablet 611c37485cd24fae9374586252925abf (table rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]) successfully deleted
I20240503 07:59:04.104588 845 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:04.106220 2404 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:04.106251 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 07:59:04.106454 779 consensus_peers.cc:485] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213): Couldn't send request to peer 3a4fb386b1bb4b7fa96abbe20f05fbed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f0a00c4bec748bc957283814ca26ae1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:04.106592 4510 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:04.106778 4152 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:04.108640 4511 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:04.109205 2404 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:04.109323 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:04.111718 438 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c reported cstate change: config changed from index 7 to 8, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
W20240503 07:59:04.126045 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 07:59:04.561826 4530 ts_tablet_manager.cc:902] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Initiating tablet copy from peer 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:59:04.562350 4530 tablet_copy_client.cc:308] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44423
I20240503 07:59:04.565603 865 tablet_copy_service.cc:140] P 5cef4772b54342fe9d418f103c30fc33: Received BeginTabletCopySession request for tablet 1f0a00c4bec748bc957283814ca26ae1 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed ({username='slave'} at 127.0.99.134:47097)
I20240503 07:59:04.565698 865 tablet_copy_service.cc:161] P 5cef4772b54342fe9d418f103c30fc33: Beginning new tablet copy session on tablet 1f0a00c4bec748bc957283814ca26ae1 from peer 3a4fb386b1bb4b7fa96abbe20f05fbed at {username='slave'} at 127.0.99.134:47097: session id = 3a4fb386b1bb4b7fa96abbe20f05fbed-1f0a00c4bec748bc957283814ca26ae1
I20240503 07:59:04.566357 865 tablet_copy_source_session.cc:205] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:04.567162 4530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0a00c4bec748bc957283814ca26ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.568881 4530 tablet_copy_client.cc:783] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:04.569039 4530 tablet_copy_client.cc:647] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:04.569973 4534 tablet_copy_client.cc:1002] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:04.620041 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 07:59:04.679736 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 07:59:05.167681 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 07:59:05.168390 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 07:59:05.331140 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 07:59:05.514734 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 07:59:06.070281 4530 tablet_copy_client.cc:523] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:06.071713 4530 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap starting.
I20240503 07:59:06.090600 4530 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:06.090831 4530 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap complete.
I20240503 07:59:06.091003 4530 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.007s
I20240503 07:59:06.091421 4530 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:06.091542 4530 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Initialized, Role: LEARNER
I20240503 07:59:06.091642 4530 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:06.091873 4530 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.092403 865 tablet_copy_service.cc:342] P 5cef4772b54342fe9d418f103c30fc33: Request end of tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-1f0a00c4bec748bc957283814ca26ae1 received from {username='slave'} at 127.0.99.134:47097
I20240503 07:59:06.092496 865 tablet_copy_service.cc:433] P 5cef4772b54342fe9d418f103c30fc33: ending tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-1f0a00c4bec748bc957283814ca26ae1 on tablet 1f0a00c4bec748bc957283814ca26ae1 with peer 3a4fb386b1bb4b7fa96abbe20f05fbed
I20240503 07:59:06.142462 2143 raft_consensus.cc:1203] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 07:59:06.167078 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 07:59:06.535387 4568 raft_consensus.cc:1050] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33: attempting to promote NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed to VOTER
I20240503 07:59:06.535835 4568 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:06.537334 2143 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:06.537350 713 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:06.537602 2404 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:06.537765 4592 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:06.537977 4594 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:06.538158 4591 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:06.540200 4592 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.540635 2143 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.541039 713 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.542729 2404 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.543756 434 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 8 to 9, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.547222 845 consensus_queue.cc:237] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:06.548314 2143 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:59:06.548377 2404 raft_consensus.cc:1261] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 07:59:06.548679 4591 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:06.548864 4592 consensus_queue.cc:1035] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:06.550335 4568 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.550702 2404 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.550702 2143 raft_consensus.cc:2932] T 1f0a00c4bec748bc957283814ca26ae1 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.551661 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f0a00c4bec748bc957283814ca26ae1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:06.552801 438 catalog_manager.cc:5484] T 1f0a00c4bec748bc957283814ca26ae1 P 164b774d3a3346bca3572f022142e17c reported cstate change: config changed from index 9 to 10, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:06.555150 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0a00c4bec748bc957283814ca26ae1 with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38956
I20240503 07:59:06.555474 4603 tablet_replica.cc:330] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:59:06.555599 4603 raft_consensus.cc:2229] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:06.555784 4603 raft_consensus.cc:2258] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:06.556353 4603 ts_tablet_manager.cc:1875] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:06.558234 4603 ts_tablet_manager.cc:1888] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 07:59:06.558295 4603 log.cc:1196] T 1f0a00c4bec748bc957283814ca26ae1 P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1f0a00c4bec748bc957283814ca26ae1
I20240503 07:59:06.558688 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 1f0a00c4bec748bc957283814ca26ae1 (table rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]) successfully deleted
I20240503 07:59:06.649407 2123 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:06.652364 2385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:06.653863 812 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:06.654832 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:06.656131 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:06.662458 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:06.671231 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:06.704747 975 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:06.706091 2273 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:06.706166 910 consensus_peers.cc:485] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dac1e203d8a47ae8cbc331ffe5f211b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:06.706274 845 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:06.706586 4655 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.706774 4654 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.708554 4529 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:06.708957 845 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:06.708982 2273 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:06.711552 441 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 6 to 7, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:06.719317 845 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:06.720598 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:06.720606 2273 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:06.720820 778 consensus_peers.cc:485] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 -> Peer 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427): Couldn't send request to peer 164b774d3a3346bca3572f022142e17c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71d661e6c1374f9282f8eff323861e1e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:06.720963 4594 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.721117 4592 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.722710 4591 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:06.723176 2273 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:06.723166 713 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:06.725220 432 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 6 to 7, NON_VOTER 164b774d3a3346bca3572f022142e17c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } } }
I20240503 07:59:06.732292 975 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:06.733597 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:06.733788 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:06.733804 910 consensus_peers.cc:485] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213): Couldn't send request to peer 3a4fb386b1bb4b7fa96abbe20f05fbed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1f5e5ffc9946639c4da0129d6d479e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:06.733964 4529 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.734170 4654 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:06.735867 4655 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:06.736261 2404 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:06.736359 713 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } } }
I20240503 07:59:06.738972 432 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 6 to 7, NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:06.767032 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 07:59:06.887113 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1f0a00c4bec748bc957283814ca26ae1: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:07.135387 4709 ts_tablet_manager.cc:902] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:59:07.135967 4709 tablet_copy_client.cc:308] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:59:07.136472 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet ae1f5e5ffc9946639c4da0129d6d479e from peer 3a4fb386b1bb4b7fa96abbe20f05fbed ({username='slave'} at 127.0.99.134:56821)
I20240503 07:59:07.136569 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet ae1f5e5ffc9946639c4da0129d6d479e from peer 3a4fb386b1bb4b7fa96abbe20f05fbed at {username='slave'} at 127.0.99.134:56821: session id = 3a4fb386b1bb4b7fa96abbe20f05fbed-ae1f5e5ffc9946639c4da0129d6d479e
I20240503 07:59:07.137362 996 tablet_copy_source_session.cc:205] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:07.138149 4709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f5e5ffc9946639c4da0129d6d479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.139853 4709 tablet_copy_client.cc:783] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:07.139984 4709 tablet_copy_client.cc:647] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:07.140925 4713 tablet_copy_client.cc:1002] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:07.149721 4714 ts_tablet_manager.cc:902] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:59:07.150246 4714 tablet_copy_client.cc:308] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:59:07.150822 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 2dac1e203d8a47ae8cbc331ffe5f211b from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:55979)
I20240503 07:59:07.150938 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 2dac1e203d8a47ae8cbc331ffe5f211b from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:55979: session id = 9e85f3485213464fb71cb81d65aedbad-2dac1e203d8a47ae8cbc331ffe5f211b
I20240503 07:59:07.151687 996 tablet_copy_source_session.cc:205] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:07.152477 4714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dac1e203d8a47ae8cbc331ffe5f211b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.154067 4714 tablet_copy_client.cc:783] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:07.154191 4714 tablet_copy_client.cc:647] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:07.155280 4718 tablet_copy_client.cc:1002] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:07.224120 4734 ts_tablet_manager.cc:902] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Initiating tablet copy from peer 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:59:07.224730 4734 tablet_copy_client.cc:308] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44423
I20240503 07:59:07.225267 865 tablet_copy_service.cc:140] P 5cef4772b54342fe9d418f103c30fc33: Received BeginTabletCopySession request for tablet 71d661e6c1374f9282f8eff323861e1e from peer 164b774d3a3346bca3572f022142e17c ({username='slave'} at 127.0.99.136:56809)
I20240503 07:59:07.225420 865 tablet_copy_service.cc:161] P 5cef4772b54342fe9d418f103c30fc33: Beginning new tablet copy session on tablet 71d661e6c1374f9282f8eff323861e1e from peer 164b774d3a3346bca3572f022142e17c at {username='slave'} at 127.0.99.136:56809: session id = 164b774d3a3346bca3572f022142e17c-71d661e6c1374f9282f8eff323861e1e
I20240503 07:59:07.226193 865 tablet_copy_source_session.cc:205] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:07.227070 4734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d661e6c1374f9282f8eff323861e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.228830 4734 tablet_copy_client.cc:783] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:07.228982 4734 tablet_copy_client.cc:647] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:07.229928 4738 tablet_copy_client.cc:1002] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:07.355995 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 07:59:07.572700 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 07:59:07.606072 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 07:59:08.108012 4015 consensus_queue.cc:579] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.198s)
I20240503 07:59:08.309279 4806 consensus_queue.cc:579] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.317s)
W20240503 07:59:08.310583 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 07:59:08.391911 4771 consensus_queue.cc:579] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.381s)
W20240503 07:59:08.585165 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 07:59:08.610253 4015 consensus_queue.cc:579] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.332s)
I20240503 07:59:08.641261 4709 tablet_copy_client.cc:523] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:08.642563 4709 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap starting.
I20240503 07:59:08.655579 4714 tablet_copy_client.cc:523] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:08.657140 4714 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:59:08.662010 4709 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:08.662262 4709 tablet_bootstrap.cc:492] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Bootstrap complete.
I20240503 07:59:08.662427 4709 ts_tablet_manager.cc:1367] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240503 07:59:08.662868 4709 raft_consensus.cc:348] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:08.663028 4709 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a4fb386b1bb4b7fa96abbe20f05fbed, State: Initialized, Role: LEARNER
I20240503 07:59:08.663131 4709 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: true } }
I20240503 07:59:08.663405 4709 ts_tablet_manager.cc:1398] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:08.663951 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-ae1f5e5ffc9946639c4da0129d6d479e received from {username='slave'} at 127.0.99.134:56821
I20240503 07:59:08.664043 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 3a4fb386b1bb4b7fa96abbe20f05fbed-ae1f5e5ffc9946639c4da0129d6d479e on tablet ae1f5e5ffc9946639c4da0129d6d479e with peer 3a4fb386b1bb4b7fa96abbe20f05fbed
I20240503 07:59:08.677135 4714 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:08.677419 4714 tablet_bootstrap.cc:492] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:59:08.677592 4714 ts_tablet_manager.cc:1367] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240503 07:59:08.678005 4714 raft_consensus.cc:348] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:08.678069 2143 raft_consensus.cc:1203] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:08.678119 4714 raft_consensus.cc:729] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:59:08.678217 4714 consensus_queue.cc:260] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:08.678592 4714 ts_tablet_manager.cc:1398] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:08.679248 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-2dac1e203d8a47ae8cbc331ffe5f211b received from {username='slave'} at 127.0.99.133:55979
I20240503 07:59:08.679335 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-2dac1e203d8a47ae8cbc331ffe5f211b on tablet 2dac1e203d8a47ae8cbc331ffe5f211b with peer 9e85f3485213464fb71cb81d65aedbad
I20240503 07:59:08.730293 4734 tablet_copy_client.cc:523] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:08.731689 4734 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Bootstrap starting.
W20240503 07:59:08.747622 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 07:59:08.749819 4734 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:08.750072 4734 tablet_bootstrap.cc:492] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Bootstrap complete.
I20240503 07:59:08.750226 4734 ts_tablet_manager.cc:1367] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.002s
I20240503 07:59:08.750593 4734 raft_consensus.cc:348] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:08.750700 4734 raft_consensus.cc:729] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Initialized, Role: LEARNER
I20240503 07:59:08.750797 4734 consensus_queue.cc:260] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: true } }
I20240503 07:59:08.751116 4734 ts_tablet_manager.cc:1398] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:08.751622 865 tablet_copy_service.cc:342] P 5cef4772b54342fe9d418f103c30fc33: Request end of tablet copy session 164b774d3a3346bca3572f022142e17c-71d661e6c1374f9282f8eff323861e1e received from {username='slave'} at 127.0.99.136:56809
I20240503 07:59:08.751695 865 tablet_copy_service.cc:433] P 5cef4772b54342fe9d418f103c30fc33: ending tablet copy session 164b774d3a3346bca3572f022142e17c-71d661e6c1374f9282f8eff323861e1e on tablet 71d661e6c1374f9282f8eff323861e1e with peer 164b774d3a3346bca3572f022142e17c
I20240503 07:59:08.964802 2012 raft_consensus.cc:1203] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 07:59:09.177899 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 07:59:09.256510 4890 raft_consensus.cc:1050] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 3a4fb386b1bb4b7fa96abbe20f05fbed to VOTER
I20240503 07:59:09.256932 4890 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:09.258309 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.258564 2143 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.258589 713 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.258824 4891 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.259030 4893 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.259167 4890 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.260941 4890 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:09.261435 2143 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:09.261793 2404 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:09.262745 713 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:09.264058 441 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 7 to 8, 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:09.330556 2404 raft_consensus.cc:1203] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:09.360599 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e"
dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42438
I20240503 07:59:09.360746 975 raft_consensus.cc:595] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:09.498416 4771 raft_consensus.cc:1050] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:59:09.498795 4771 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:09.500300 2012 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.500319 2273 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.500362 843 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.500695 4891 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.500890 4890 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.501042 4893 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.503270 4893 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.503623 2012 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.503763 843 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.506178 434 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 7 to 8, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.506217 2273 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.509302 975 consensus_queue.cc:237] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:09.510454 2012 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:09.510525 2273 raft_consensus.cc:1261] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:09.510797 4891 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:09.510989 4771 consensus_queue.cc:1035] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:09.512374 4893 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.512787 2012 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.514170 2273 raft_consensus.cc:2932] T 2dac1e203d8a47ae8cbc331ffe5f211b P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:09.514690 441 catalog_manager.cc:5484] T 2dac1e203d8a47ae8cbc331ffe5f211b P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:09.516182 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dac1e203d8a47ae8cbc331ffe5f211b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:09.517149 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dac1e203d8a47ae8cbc331ffe5f211b with delete_type TABLET_DATA_TOMBSTONED (TS 5cef4772b54342fe9d418f103c30fc33 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34178
I20240503 07:59:09.517524 4930 tablet_replica.cc:330] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: stopping tablet replica
I20240503 07:59:09.517792 4930 raft_consensus.cc:2229] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:09.518033 4930 raft_consensus.cc:2258] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:09.518692 4930 ts_tablet_manager.cc:1875] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:09.520391 4930 ts_tablet_manager.cc:1888] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:09.520450 4930 log.cc:1196] T 2dac1e203d8a47ae8cbc331ffe5f211b P 5cef4772b54342fe9d418f103c30fc33: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2dac1e203d8a47ae8cbc331ffe5f211b
I20240503 07:59:09.520941 427 catalog_manager.cc:4830] TS 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423): tablet 2dac1e203d8a47ae8cbc331ffe5f211b (table rebalance_test_table_4 [id=e40382a53e8d42c6885341fc4a54ebd0]) successfully deleted
I20240503 07:59:09.583416 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e"
dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42466
I20240503 07:59:09.583560 975 raft_consensus.cc:595] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:09.601281 975 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:59:09.602742 2012 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:09.602818 843 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:09.602888 911 consensus_peers.cc:485] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer b8f135eb22964d4a8d8300956bce8017 (127.0.99.135:35843): Couldn't send request to peer b8f135eb22964d4a8d8300956bce8017. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33a8cbbafd86402a9a3bf9e730b60c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:09.603207 4771 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:09.603420 4891 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:09.605253 4890 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:59:09.605669 843 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:59:09.607079 2012 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:59:09.608194 441 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 6 to 7, NON_VOTER b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } } }
I20240503 07:59:09.623176 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e"
dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42514
I20240503 07:59:09.623296 975 raft_consensus.cc:595] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:09.686981 4891 raft_consensus.cc:979] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: : Instructing follower 7a36d94f14d04d74b632bc75d1779bc3 to start an election
I20240503 07:59:09.687101 4891 raft_consensus.cc:1067] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Signalling peer 7a36d94f14d04d74b632bc75d1779bc3 to start an election
I20240503 07:59:09.687544 713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e"
dest_uuid: "7a36d94f14d04d74b632bc75d1779bc3"
from {username='slave'} at 127.0.99.132:34477
I20240503 07:59:09.687702 713 raft_consensus.cc:482] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:09.687770 713 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:09.688695 713 raft_consensus.cc:504] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:09.689199 713 leader_election.cc:288] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 election: Requested vote from peers 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363), 164b774d3a3346bca3572f022142e17c (127.0.99.136:45427), 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213)
I20240503 07:59:09.689498 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e"
I20240503 07:59:09.689603 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
I20240503 07:59:09.689653 975 raft_consensus.cc:3032] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:09.689611 2404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f5e5ffc9946639c4da0129d6d479e" candidate_uuid: "7a36d94f14d04d74b632bc75d1779bc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "164b774d3a3346bca3572f022142e17c"
I20240503 07:59:09.689693 975 raft_consensus.cc:729] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80bd2b9fafaf44bcb8c0f1b9f408841e, State: Running, Role: LEADER
I20240503 07:59:09.689726 2143 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:09.689742 2404 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:09.689836 975 consensus_queue.cc:260] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:09.690016 975 raft_consensus.cc:3037] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:09.690428 2404 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 2.
I20240503 07:59:09.690673 2143 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 2.
I20240503 07:59:09.690891 975 raft_consensus.cc:2454] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a36d94f14d04d74b632bc75d1779bc3 in term 2.
I20240503 07:59:09.691067 648 leader_election.cc:302] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 164b774d3a3346bca3572f022142e17c, 3a4fb386b1bb4b7fa96abbe20f05fbed, 7a36d94f14d04d74b632bc75d1779bc3; no voters:
I20240503 07:59:09.691267 4805 raft_consensus.cc:2781] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:09.691426 4805 raft_consensus.cc:686] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Becoming Leader. State: Replica: 7a36d94f14d04d74b632bc75d1779bc3, State: Running, Role: LEADER
I20240503 07:59:09.691573 4805 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:09.693197 432 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: term changed from 1 to 2, leader changed from 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) to 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:09.712926 4891 consensus_queue.cc:579] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.138s)
W20240503 07:59:09.714111 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 07:59:09.718122 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 07:59:09.877686 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 07:59:09.878391 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 611c37485cd24fae9374586252925abf: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:09.915458 4998 raft_consensus.cc:1050] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33: attempting to promote NON_VOTER 164b774d3a3346bca3572f022142e17c to VOTER
I20240503 07:59:09.915935 4998 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:59:09.917325 2404 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 LEARNER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.917413 2273 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.917474 713 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:09.917740 5001 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.917927 5000 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.918079 4998 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:09.920601 5000 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.921113 2273 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.921205 713 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.922298 2404 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.923528 432 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 7 to 8, 164b774d3a3346bca3572f022142e17c (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.927083 843 consensus_queue.cc:237] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } }
I20240503 07:59:09.928094 2273 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:09.928165 2404 raft_consensus.cc:1261] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:09.928457 5000 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:09.928619 4998 consensus_queue.cc:1035] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:09.930037 4594 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.930555 2404 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P 164b774d3a3346bca3572f022142e17c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.931486 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71d661e6c1374f9282f8eff323861e1e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:09.931800 2273 raft_consensus.cc:2932] T 71d661e6c1374f9282f8eff323861e1e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } }
I20240503 07:59:09.932286 441 catalog_manager.cc:5484] T 71d661e6c1374f9282f8eff323861e1e P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 8 to 9, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:09.934556 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d661e6c1374f9282f8eff323861e1e with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38956
I20240503 07:59:09.934971 5006 tablet_replica.cc:330] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:59:09.935111 5006 raft_consensus.cc:2229] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:09.935290 5006 raft_consensus.cc:2258] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:09.936074 5006 ts_tablet_manager.cc:1875] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:09.937803 5006 ts_tablet_manager.cc:1888] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:09.937862 5006 log.cc:1196] T 71d661e6c1374f9282f8eff323861e1e P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/71d661e6c1374f9282f8eff323861e1e
I20240503 07:59:09.938359 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 71d661e6c1374f9282f8eff323861e1e (table rebalance_test_table_3 [id=4d6fc31d9d3b4fb6b8ef40d2fa075652]) successfully deleted
W20240503 07:59:09.972318 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 07:59:09.993801 5007 ts_tablet_manager.cc:902] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Initiating tablet copy from peer 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363)
I20240503 07:59:09.994400 5007 tablet_copy_client.cc:308] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39363
I20240503 07:59:09.994895 996 tablet_copy_service.cc:140] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Received BeginTabletCopySession request for tablet 33a8cbbafd86402a9a3bf9e730b60c0e from peer b8f135eb22964d4a8d8300956bce8017 ({username='slave'} at 127.0.99.135:32837)
I20240503 07:59:09.995019 996 tablet_copy_service.cc:161] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Beginning new tablet copy session on tablet 33a8cbbafd86402a9a3bf9e730b60c0e from peer b8f135eb22964d4a8d8300956bce8017 at {username='slave'} at 127.0.99.135:32837: session id = b8f135eb22964d4a8d8300956bce8017-33a8cbbafd86402a9a3bf9e730b60c0e
I20240503 07:59:09.995754 996 tablet_copy_source_session.cc:205] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:09.996559 5007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a8cbbafd86402a9a3bf9e730b60c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.998310 5007 tablet_copy_client.cc:783] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:09.998461 5007 tablet_copy_client.cc:647] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:09.999390 5011 tablet_copy_client.cc:1002] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:10.070358 2143 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:10.071820 2143 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:10.072306 4805 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:10.072507 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 07:59:10.072644 714 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 07:59:10.073040 4839 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:10.073360 4839 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:10.075500 5034 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:10.075919 714 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
W20240503 07:59:10.076272 2077 consensus_peers.cc:485] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 611c37485cd24fae9374586252925abf. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.078421 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:10.078878 434 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 11 to 12, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.088376 975 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:10.089004 4805 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:10.091912 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:10.093163 4979 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:10.143815 4594 consensus_queue.cc:579] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Leader has been unable to successfully communicate with peer 27d51a644d914549b3dfb3f104d396e8 for more than 30 seconds (30.355s)
W20240503 07:59:10.145112 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 07:59:10.505456 5078 ts_tablet_manager.cc:902] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213)
I20240503 07:59:10.505995 5078 tablet_copy_client.cc:308] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:34213
I20240503 07:59:10.509479 2163 tablet_copy_service.cc:140] P 3a4fb386b1bb4b7fa96abbe20f05fbed: Received BeginTabletCopySession request for tablet 611c37485cd24fae9374586252925abf from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:56597)
I20240503 07:59:10.509613 2163 tablet_copy_service.cc:161] P 3a4fb386b1bb4b7fa96abbe20f05fbed: Beginning new tablet copy session on tablet 611c37485cd24fae9374586252925abf from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:56597: session id = 9e85f3485213464fb71cb81d65aedbad-611c37485cd24fae9374586252925abf
I20240503 07:59:10.510951 2163 tablet_copy_source_session.cc:205] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:10.512132 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611c37485cd24fae9374586252925abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.513839 5078 tablet_copy_client.cc:783] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:10.513998 5078 tablet_copy_client.cc:647] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:10.515182 5082 tablet_copy_client.cc:1002] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:10.620651 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 07:59:10.716289 714 consensus_queue.cc:237] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } }
I20240503 07:59:10.717394 2404 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:10.717483 2143 raft_consensus.cc:1261] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Refusing update from remote peer 7a36d94f14d04d74b632bc75d1779bc3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:10.717741 5100 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:10.717947 5051 consensus_queue.cc:1035] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:10.719425 5100 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:10.719919 2143 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:10.720083 2404 raft_consensus.cc:2932] T ae1f5e5ffc9946639c4da0129d6d479e P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } }
I20240503 07:59:10.721282 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae1f5e5ffc9946639c4da0129d6d479e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:10.722106 435 catalog_manager.cc:5484] T ae1f5e5ffc9946639c4da0129d6d479e P 7a36d94f14d04d74b632bc75d1779bc3 reported cstate change: config changed from index 8 to 10, VOTER 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "7a36d94f14d04d74b632bc75d1779bc3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:10.724625 956 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1f5e5ffc9946639c4da0129d6d479e with delete_type TABLET_DATA_TOMBSTONED (TS 80bd2b9fafaf44bcb8c0f1b9f408841e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43524
I20240503 07:59:10.725075 5105 tablet_replica.cc:330] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: stopping tablet replica
I20240503 07:59:10.725596 5105 raft_consensus.cc:2229] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.725755 5105 raft_consensus.cc:2258] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.726289 5105 ts_tablet_manager.cc:1875] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:10.727972 5105 ts_tablet_manager.cc:1888] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 07:59:10.728032 5105 log.cc:1196] T ae1f5e5ffc9946639c4da0129d6d479e P 80bd2b9fafaf44bcb8c0f1b9f408841e: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae1f5e5ffc9946639c4da0129d6d479e
I20240503 07:59:10.728510 428 catalog_manager.cc:4830] TS 80bd2b9fafaf44bcb8c0f1b9f408841e (127.0.99.132:39363): tablet ae1f5e5ffc9946639c4da0129d6d479e (table rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]) successfully deleted
W20240503 07:59:10.772342 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 07:59:10.999699 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 07:59:11.499677 5007 tablet_copy_client.cc:523] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.500968 5007 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Bootstrap starting.
I20240503 07:59:11.519697 5007 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.519954 5007 tablet_bootstrap.cc:492] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Bootstrap complete.
I20240503 07:59:11.520112 5007 ts_tablet_manager.cc:1367] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240503 07:59:11.520545 5007 raft_consensus.cc:348] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:59:11.520659 5007 raft_consensus.cc:729] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f135eb22964d4a8d8300956bce8017, State: Initialized, Role: LEARNER
I20240503 07:59:11.520749 5007 consensus_queue.cc:260] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:59:11.521034 5007 ts_tablet_manager.cc:1398] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:11.521608 996 tablet_copy_service.cc:342] P 80bd2b9fafaf44bcb8c0f1b9f408841e: Request end of tablet copy session b8f135eb22964d4a8d8300956bce8017-33a8cbbafd86402a9a3bf9e730b60c0e received from {username='slave'} at 127.0.99.135:32837
I20240503 07:59:11.521699 996 tablet_copy_service.cc:433] P 80bd2b9fafaf44bcb8c0f1b9f408841e: ending tablet copy session b8f135eb22964d4a8d8300956bce8017-33a8cbbafd86402a9a3bf9e730b60c0e on tablet 33a8cbbafd86402a9a3bf9e730b60c0e with peer b8f135eb22964d4a8d8300956bce8017
W20240503 07:59:11.808460 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 07:59:11.857213 2274 raft_consensus.cc:1203] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:12.015467 5078 tablet_copy_client.cc:523] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.016726 5078 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:59:12.039170 5078 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.039412 5078 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:59:12.039562 5078 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.005s
I20240503 07:59:12.039943 5078 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:12.040056 5078 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:59:12.040148 5078 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:12.040477 5078 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.041069 2163 tablet_copy_service.cc:342] P 3a4fb386b1bb4b7fa96abbe20f05fbed: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-611c37485cd24fae9374586252925abf received from {username='slave'} at 127.0.99.133:56597
I20240503 07:59:12.041159 2163 tablet_copy_service.cc:433] P 3a4fb386b1bb4b7fa96abbe20f05fbed: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-611c37485cd24fae9374586252925abf on tablet 611c37485cd24fae9374586252925abf with peer 9e85f3485213464fb71cb81d65aedbad
W20240503 07:59:12.068570 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 07:59:12.301388 2012 raft_consensus.cc:1203] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240503 07:59:12.326400 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 07:59:12.357971 5213 raft_consensus.cc:1050] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e: attempting to promote NON_VOTER b8f135eb22964d4a8d8300956bce8017 to VOTER
I20240503 07:59:12.358366 5213 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:59:12.359704 2274 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 LEARNER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:12.359822 843 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:12.359953 2012 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:12.360198 5212 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:12.360384 5144 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:12.360594 5215 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:12.362336 5144 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.362768 2012 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.364898 441 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e reported cstate change: config changed from index 7 to 8, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.365314 843 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.365763 2273 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f135eb22964d4a8d8300956bce8017 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.369351 975 consensus_queue.cc:237] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } }
I20240503 07:59:12.370409 2273 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:12.370497 2012 raft_consensus.cc:1261] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Refusing update from remote peer 80bd2b9fafaf44bcb8c0f1b9f408841e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:12.370735 5212 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:12.370983 5215 consensus_queue.cc:1035] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:12.372117 5212 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 80bd2b9fafaf44bcb8c0f1b9f408841e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.372507 2273 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.372998 2012 raft_consensus.cc:2932] T 33a8cbbafd86402a9a3bf9e730b60c0e P 9e85f3485213464fb71cb81d65aedbad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.374256 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33a8cbbafd86402a9a3bf9e730b60c0e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.374411 435 catalog_manager.cc:5484] T 33a8cbbafd86402a9a3bf9e730b60c0e P b8f135eb22964d4a8d8300956bce8017 reported cstate change: config changed from index 8 to 9, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: false } } }
I20240503 07:59:12.376638 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a8cbbafd86402a9a3bf9e730b60c0e with delete_type TABLET_DATA_TOMBSTONED (TS 5cef4772b54342fe9d418f103c30fc33 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34178
I20240503 07:59:12.376976 5225 tablet_replica.cc:330] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: stopping tablet replica
I20240503 07:59:12.377094 5225 raft_consensus.cc:2229] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.377245 5225 raft_consensus.cc:2258] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.377737 5225 ts_tablet_manager.cc:1875] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.379346 5225 ts_tablet_manager.cc:1888] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:12.379405 5225 log.cc:1196] T 33a8cbbafd86402a9a3bf9e730b60c0e P 5cef4772b54342fe9d418f103c30fc33: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/33a8cbbafd86402a9a3bf9e730b60c0e
I20240503 07:59:12.379828 427 catalog_manager.cc:4830] TS 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423): tablet 33a8cbbafd86402a9a3bf9e730b60c0e (table rebalance_test_table_1 [id=3346aee7b032427b9f0804fb25fca756]) successfully deleted
W20240503 07:59:12.555068 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 07:59:12.921768 5253 raft_consensus.cc:1050] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:59:12.922214 5253 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:12.923511 2012 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 LEARNER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 07:59:12.923539 714 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 07:59:12.923605 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 07:59:12.923938 5256 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:12.924135 5253 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:12.924320 5255 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:12.926146 5255 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.926589 2011 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.926849 714 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.928256 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.929158 432 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 12 to 13, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.932138 2143 consensus_queue.cc:237] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:12.933411 2011 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 07:59:12.933712 2404 raft_consensus.cc:1261] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Refusing update from remote peer 3a4fb386b1bb4b7fa96abbe20f05fbed: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 07:59:12.933775 5256 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:12.934055 5255 consensus_queue.cc:1035] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:12.935127 5256 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.935581 2011 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.935789 2404 raft_consensus.cc:2932] T 611c37485cd24fae9374586252925abf P 164b774d3a3346bca3572f022142e17c [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.936571 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 611c37485cd24fae9374586252925abf with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.937928 441 catalog_manager.cc:5484] T 611c37485cd24fae9374586252925abf P 9e85f3485213464fb71cb81d65aedbad reported cstate change: config changed from index 13 to 14, VOTER 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:12.940264 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 611c37485cd24fae9374586252925abf with delete_type TABLET_DATA_TOMBSTONED (TS 7a36d94f14d04d74b632bc75d1779bc3 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:38956
I20240503 07:59:12.940606 5265 tablet_replica.cc:330] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: stopping tablet replica
I20240503 07:59:12.940729 5265 raft_consensus.cc:2229] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.940960 5265 raft_consensus.cc:2258] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.941601 5265 ts_tablet_manager.cc:1875] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.943289 5265 ts_tablet_manager.cc:1888] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 07:59:12.943359 5265 log.cc:1196] T 611c37485cd24fae9374586252925abf P 7a36d94f14d04d74b632bc75d1779bc3: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/611c37485cd24fae9374586252925abf
I20240503 07:59:12.943828 427 catalog_manager.cc:4830] TS 7a36d94f14d04d74b632bc75d1779bc3 (127.0.99.130:38831): tablet 611c37485cd24fae9374586252925abf (table rebalance_test_table_2 [id=65ecf2b2345f40e297d40a7e2ade7a99]) successfully deleted
I20240503 07:59:13.091339 843 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:13.092733 2404 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 07:59:13.092978 2143 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 07:59:13.093123 5279 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:13.093360 5278 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:13.094856 5219 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:13.095250 2404 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
I20240503 07:59:13.097316 435 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 12 to 13, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New cstate: current_term: 3 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.098830 2143 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } } }
W20240503 07:59:13.100675 779 consensus_peers.cc:485] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799): Couldn't send request to peer 9e85f3485213464fb71cb81d65aedbad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c2bc8609e4945148a50dde608502f67. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:13.277282 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 07:59:13.397970 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 07:59:13.570642 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eca5e6693b24d0e8f3d9c37a62a1bca: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:13.587327 5302 ts_tablet_manager.cc:902] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Initiating tablet copy from peer 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423)
I20240503 07:59:13.587790 5302 tablet_copy_client.cc:308] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44423
I20240503 07:59:13.591101 865 tablet_copy_service.cc:140] P 5cef4772b54342fe9d418f103c30fc33: Received BeginTabletCopySession request for tablet 4c2bc8609e4945148a50dde608502f67 from peer 9e85f3485213464fb71cb81d65aedbad ({username='slave'} at 127.0.99.133:47299)
I20240503 07:59:13.591207 865 tablet_copy_service.cc:161] P 5cef4772b54342fe9d418f103c30fc33: Beginning new tablet copy session on tablet 4c2bc8609e4945148a50dde608502f67 from peer 9e85f3485213464fb71cb81d65aedbad at {username='slave'} at 127.0.99.133:47299: session id = 9e85f3485213464fb71cb81d65aedbad-4c2bc8609e4945148a50dde608502f67
I20240503 07:59:13.591933 865 tablet_copy_source_session.cc:205] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.592742 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2bc8609e4945148a50dde608502f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.594740 5302 tablet_copy_client.cc:783] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.594877 5302 tablet_copy_client.cc:647] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.595937 5306 tablet_copy_client.cc:1002] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:13.969157 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 07:59:14.297669 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 07:59:14.567833 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 07:59:14.743129 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 07:59:14.797320 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 07:59:15.096276 5302 tablet_copy_client.cc:523] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.097669 5302 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap starting.
I20240503 07:59:15.116815 5302 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.117100 5302 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Bootstrap complete.
I20240503 07:59:15.117275 5302 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 07:59:15.117687 5302 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:15.117798 5302 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9e85f3485213464fb71cb81d65aedbad, State: Initialized, Role: LEARNER
I20240503 07:59:15.117882 5302 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: true } }
I20240503 07:59:15.118186 5302 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:15.118669 865 tablet_copy_service.cc:342] P 5cef4772b54342fe9d418f103c30fc33: Request end of tablet copy session 9e85f3485213464fb71cb81d65aedbad-4c2bc8609e4945148a50dde608502f67 received from {username='slave'} at 127.0.99.133:47299
I20240503 07:59:15.118743 865 tablet_copy_service.cc:433] P 5cef4772b54342fe9d418f103c30fc33: ending tablet copy session 9e85f3485213464fb71cb81d65aedbad-4c2bc8609e4945148a50dde608502f67 on tablet 4c2bc8609e4945148a50dde608502f67 with peer 9e85f3485213464fb71cb81d65aedbad
W20240503 07:59:15.168697 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 07:59:15.456346 2011 raft_consensus.cc:1203] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 LEARNER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
W20240503 07:59:15.743521 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 07:59:15.832782 649 consensus_peers.cc:485] T 24e17467ff094742bf72f568c0953973 P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 07:59:15.862592 649 consensus_peers.cc:485] T f42fb6ac38fc4b4782814cae0891b0cc P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 07:59:16.062986 5382 raft_consensus.cc:1050] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: attempting to promote NON_VOTER 9e85f3485213464fb71cb81d65aedbad to VOTER
I20240503 07:59:16.063452 5382 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:16.064934 2404 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 07:59:16.064972 2142 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 07:59:16.065076 2011 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 LEARNER]: Refusing update from remote peer 5cef4772b54342fe9d418f103c30fc33: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 07:59:16.065513 5381 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:16.065742 5384 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:16.065913 5320 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:16.067961 5320 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.069447 2142 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.069672 2404 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.070516 434 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 reported cstate change: config changed from index 13 to 14, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "5cef4772b54342fe9d418f103c30fc33" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.071295 2011 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 9e85f3485213464fb71cb81d65aedbad (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.207849 843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42500
I20240503 07:59:16.208006 843 raft_consensus.cc:595] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Received request to transfer leadership
I20240503 07:59:16.418665 843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42526
I20240503 07:59:16.418798 843 raft_consensus.cc:595] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Received request to transfer leadership
I20240503 07:59:16.494992 5381 raft_consensus.cc:979] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: : Instructing follower 164b774d3a3346bca3572f022142e17c to start an election
I20240503 07:59:16.495137 5384 raft_consensus.cc:1067] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Signalling peer 164b774d3a3346bca3572f022142e17c to start an election
I20240503 07:59:16.495724 2404 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67"
dest_uuid: "164b774d3a3346bca3572f022142e17c"
from {username='slave'} at 127.0.99.131:41741
I20240503 07:59:16.495908 2404 raft_consensus.cc:482] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:16.495976 2404 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:16.496884 2404 raft_consensus.cc:504] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:16.498061 2404 leader_election.cc:288] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [CANDIDATE]: Term 4 election: Requested vote from peers 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423), 3a4fb386b1bb4b7fa96abbe20f05fbed (127.0.99.134:34213), 9e85f3485213464fb71cb81d65aedbad (127.0.99.133:33799)
I20240503 07:59:16.498279 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "164b774d3a3346bca3572f022142e17c" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "5cef4772b54342fe9d418f103c30fc33"
I20240503 07:59:16.498394 843 raft_consensus.cc:3032] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:16.498430 843 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cef4772b54342fe9d418f103c30fc33, State: Running, Role: LEADER
I20240503 07:59:16.498587 843 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:16.498781 843 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:16.499542 843 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 164b774d3a3346bca3572f022142e17c in term 4.
I20240503 07:59:16.502557 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "164b774d3a3346bca3572f022142e17c" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed"
I20240503 07:59:16.502689 2142 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:16.503338 2011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2bc8609e4945148a50dde608502f67" candidate_uuid: "164b774d3a3346bca3572f022142e17c" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "9e85f3485213464fb71cb81d65aedbad"
I20240503 07:59:16.503521 2011 raft_consensus.cc:3037] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:16.503587 2142 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 164b774d3a3346bca3572f022142e17c in term 4.
I20240503 07:59:16.503875 2339 leader_election.cc:302] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 164b774d3a3346bca3572f022142e17c, 3a4fb386b1bb4b7fa96abbe20f05fbed, 5cef4772b54342fe9d418f103c30fc33; no voters:
I20240503 07:59:16.504359 2011 raft_consensus.cc:2454] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 164b774d3a3346bca3572f022142e17c in term 4.
I20240503 07:59:16.504578 5388 raft_consensus.cc:2781] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:16.504760 5388 raft_consensus.cc:686] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 4 LEADER]: Becoming Leader. State: Replica: 164b774d3a3346bca3572f022142e17c, State: Running, Role: LEADER
I20240503 07:59:16.505055 5388 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:16.507310 441 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c reported cstate change: term changed from 3 to 4, leader changed from 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) to 164b774d3a3346bca3572f022142e17c (127.0.99.136). New cstate: current_term: 4 leader_uuid: "164b774d3a3346bca3572f022142e17c" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:16.581133 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 07:59:16.581935 427 catalog_manager.cc:4573] TS 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af4caa7f67b9473192241d4030f07f99: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111)
I20240503 07:59:16.923166 2142 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 4 FOLLOWER]: Refusing update from remote peer 164b774d3a3346bca3572f022142e17c: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 07:59:16.923655 5388 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 07:59:16.946409 2011 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 4 FOLLOWER]: Refusing update from remote peer 164b774d3a3346bca3572f022142e17c: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 07:59:16.946841 5388 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 07:59:16.950004 843 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 4 FOLLOWER]: Refusing update from remote peer 164b774d3a3346bca3572f022142e17c: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 07:59:16.950649 5388 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 07:59:16.955017 2404 consensus_queue.cc:237] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 14, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } }
I20240503 07:59:16.956050 2011 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 4 FOLLOWER]: Refusing update from remote peer 164b774d3a3346bca3572f022142e17c: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240503 07:59:16.956080 2142 raft_consensus.cc:1261] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 4 FOLLOWER]: Refusing update from remote peer 164b774d3a3346bca3572f022142e17c: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240503 07:59:16.956373 5388 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:16.956537 5421 consensus_queue.cc:1035] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:16.958115 5439 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 164b774d3a3346bca3572f022142e17c [term 4 LEADER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.958536 2142 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.958583 2011 raft_consensus.cc:2932] T 4c2bc8609e4945148a50dde608502f67 P 9e85f3485213464fb71cb81d65aedbad [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.960404 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c2bc8609e4945148a50dde608502f67 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.960707 432 catalog_manager.cc:5484] T 4c2bc8609e4945148a50dde608502f67 P 3a4fb386b1bb4b7fa96abbe20f05fbed reported cstate change: config changed from index 14 to 16, VOTER 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131) evicted. New cstate: current_term: 4 leader_uuid: "164b774d3a3346bca3572f022142e17c" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3a4fb386b1bb4b7fa96abbe20f05fbed" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34213 } attrs { promote: false } } peers { permanent_uuid: "164b774d3a3346bca3572f022142e17c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 45427 } attrs { promote: false } } peers { permanent_uuid: "9e85f3485213464fb71cb81d65aedbad" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33799 } attrs { promote: false } } }
I20240503 07:59:16.962951 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2bc8609e4945148a50dde608502f67 with delete_type TABLET_DATA_TOMBSTONED (TS 5cef4772b54342fe9d418f103c30fc33 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:34178
I20240503 07:59:16.963263 5443 tablet_replica.cc:330] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: stopping tablet replica
I20240503 07:59:16.963387 5443 raft_consensus.cc:2229] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.963495 5443 raft_consensus.cc:2258] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.964130 5443 ts_tablet_manager.cc:1875] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.966591 5443 ts_tablet_manager.cc:1888] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.15
I20240503 07:59:16.966670 5443 log.cc:1196] T 4c2bc8609e4945148a50dde608502f67 P 5cef4772b54342fe9d418f103c30fc33: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c2bc8609e4945148a50dde608502f67
I20240503 07:59:16.967257 427 catalog_manager.cc:4830] TS 5cef4772b54342fe9d418f103c30fc33 (127.0.99.131:44423): tablet 4c2bc8609e4945148a50dde608502f67 (table rebalance_test_table_0 [id=4f8dbea637c2414f95f05cb0f6d0abb8]) successfully deleted
W20240503 07:59:16.984786 649 consensus_peers.cc:485] T 6e96426aaa194d8cb60e03a31abfd83d P 7a36d94f14d04d74b632bc75d1779bc3 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 07:59:17.088856 911 consensus_peers.cc:485] T 7f6c2af6b9764ad1b464973bc96853ec P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 07:59:17.101716 2123 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.102450 2385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.103698 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.104625 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.106292 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.112076 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.131598 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.207441 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.207665 2385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.210014 2123 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 07:59:17.210335 911 consensus_peers.cc:485] T ac39a827f881410c83898fe3a26e34f1 P 80bd2b9fafaf44bcb8c0f1b9f408841e -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 07:59:17.214491 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.227919 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.230227 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.237912 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.270040 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:36481
--local_ip_for_outbound_sockets=127.0.99.129
--tserver_master_addrs=127.0.99.190:37543
--webserver_port=43113
--webserver_interface=127.0.99.129
--builtin_ntp_servers=127.0.99.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:17.376351 5522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:17.376561 5522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:17.376623 5522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:17.379560 5522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:17.379631 5522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:17.379715 5522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:17.383287 5522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:36481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=43113
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:37543
--never_fsync=true
--heap_profile_path=/tmp/kudu.5522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:17.384176 5522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:17.385473 5522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:17.394223 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:17.394753 5527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:17.395388 5528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:17.396014 5522 server_base.cc:1034] running on GCE node
I20240503 07:59:17.396457 5522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:17.397202 5522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:17.398365 5522 hybrid_clock.cc:648] HybridClock initialized: now 1714723157398341 us; error 35 us; skew 500 ppm
I20240503 07:59:17.400513 5522 webserver.cc:460] Webserver started at http://127.0.99.129:43113/ using document root <none> and password file <none>
I20240503 07:59:17.401140 5522 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:17.401225 5522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:17.405212 5522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:17.408138 5536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:17.409127 5522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:17.409224 5522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "27d51a644d914549b3dfb3f104d396e8"
format_stamp: "Formatted at 2024-05-03 07:58:27 on dist-test-slave-11df"
I20240503 07:59:17.409592 5522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:17.422204 5522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:17.422775 5522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:17.422900 5522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:17.423468 5522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:17.424587 5546 ts_tablet_manager.cc:516] Loading tablet metadata (0/11 complete)
I20240503 07:59:17.433419 5522 ts_tablet_manager.cc:559] Loaded tablet metadata (11 total tablets, 11 live tablets)
I20240503 07:59:17.433490 5522 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:17.433552 5522 ts_tablet_manager.cc:574] Registering tablets (0/11 complete)
I20240503 07:59:17.435789 5546 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.439478 5522 ts_tablet_manager.cc:590] Registered 11 tablets
I20240503 07:59:17.439527 5522 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20240503 07:59:17.450980 5522 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36481
I20240503 07:59:17.451184 5653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36481 every 8 connection(s)
I20240503 07:59:17.452060 5522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:17.452390 5546 log.cc:824] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:17.455544 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 5522
I20240503 07:59:17.462283 5654 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37543
I20240503 07:59:17.462705 5654 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:17.463389 5654 heartbeater.cc:511] Master 127.0.99.190:37543 requested a full tablet report, sending...
I20240503 07:59:17.465441 438 ts_manager.cc:194] Re-registered known tserver with Master: 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481)
I20240503 07:59:17.467936 438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:37741
I20240503 07:59:17.477214 5546 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.477603 5546 tablet_bootstrap.cc:492] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.478411 5546 ts_tablet_manager.cc:1367] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.043s user 0.029s sys 0.009s
I20240503 07:59:17.480368 5546 raft_consensus.cc:348] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:59:17.480670 5546 raft_consensus.cc:729] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.481082 5546 consensus_queue.cc:260] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:59:17.481731 5654 heartbeater.cc:503] Master 127.0.99.190:37543 was elected leader, sending a full tablet report...
I20240503 07:59:17.481853 5546 ts_tablet_manager.cc:1398] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 07:59:17.482086 5546 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.508165 5546 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.508472 5546 tablet_bootstrap.cc:492] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.509145 5546 ts_tablet_manager.cc:1367] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240503 07:59:17.509618 5546 raft_consensus.cc:348] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.509733 5546 raft_consensus.cc:729] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.509846 5546 consensus_queue.cc:260] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.510082 5546 ts_tablet_manager.cc:1398] T 611c37485cd24fae9374586252925abf P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.510216 5546 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.532922 5546 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.533257 5546 tablet_bootstrap.cc:492] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.533954 5546 ts_tablet_manager.cc:1367] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20240503 07:59:17.534365 5546 raft_consensus.cc:348] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:59:17.534471 5546 raft_consensus.cc:729] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.534596 5546 consensus_queue.cc:260] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:59:17.534857 5546 ts_tablet_manager.cc:1398] T 6e96426aaa194d8cb60e03a31abfd83d P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.535104 5546 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.559473 5546 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.559770 5546 tablet_bootstrap.cc:492] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.560441 5546 ts_tablet_manager.cc:1367] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.009s
I20240503 07:59:17.560817 5546 raft_consensus.cc:348] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.560916 5546 raft_consensus.cc:729] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.561043 5546 consensus_queue.cc:260] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.561283 5546 ts_tablet_manager.cc:1398] T ac39a827f881410c83898fe3a26e34f1 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.561417 5546 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.586048 5546 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.586421 5546 tablet_bootstrap.cc:492] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.587285 5546 ts_tablet_manager.cc:1367] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.003s
I20240503 07:59:17.587716 5546 raft_consensus.cc:348] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.587834 5546 raft_consensus.cc:729] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.587978 5546 consensus_queue.cc:260] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } }
I20240503 07:59:17.588256 5546 ts_tablet_manager.cc:1398] T 97138ab5e3394c41b83382a22df0a198 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.588472 5546 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.601171 5608 raft_consensus.cc:3037] T 7f6c2af6b9764ad1b464973bc96853ec P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.615052 5546 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.615386 5546 tablet_bootstrap.cc:492] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.616071 5546 ts_tablet_manager.cc:1367] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20240503 07:59:17.616425 5546 raft_consensus.cc:348] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:59:17.616519 5546 raft_consensus.cc:729] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.616633 5546 consensus_queue.cc:260] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } }
I20240503 07:59:17.616878 5546 ts_tablet_manager.cc:1398] T f42fb6ac38fc4b4782814cae0891b0cc P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.617029 5546 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.643791 5546 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.644142 5546 tablet_bootstrap.cc:492] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.645104 5546 ts_tablet_manager.cc:1367] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20240503 07:59:17.645553 5546 raft_consensus.cc:348] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.645658 5546 raft_consensus.cc:729] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.645783 5546 consensus_queue.cc:260] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.646034 5546 ts_tablet_manager.cc:1398] T 24e17467ff094742bf72f568c0953973 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.646175 5546 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.670873 2385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.673233 5588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.673337 2123 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.684954 5546 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.685354 5546 tablet_bootstrap.cc:492] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.686647 5546 ts_tablet_manager.cc:1367] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.004s
I20240503 07:59:17.687198 5546 raft_consensus.cc:348] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.687337 5546 raft_consensus.cc:729] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.687486 5546 consensus_queue.cc:260] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.687877 5546 ts_tablet_manager.cc:1398] T af4caa7f67b9473192241d4030f07f99 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.688036 5546 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.699009 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.699685 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.714614 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.715849 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.716596 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.738734 5546 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.739136 5546 tablet_bootstrap.cc:492] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.740172 5546 ts_tablet_manager.cc:1367] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.014s
I20240503 07:59:17.740612 5546 raft_consensus.cc:348] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.740736 5546 raft_consensus.cc:729] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.740877 5546 consensus_queue.cc:260] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } }
I20240503 07:59:17.741169 5546 ts_tablet_manager.cc:1398] T 4c2bc8609e4945148a50dde608502f67 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.741313 5546 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.769692 5546 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.770059 5546 tablet_bootstrap.cc:492] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.771111 5546 ts_tablet_manager.cc:1367] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.003s
I20240503 07:59:17.771533 5546 raft_consensus.cc:348] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:59:17.771649 5546 raft_consensus.cc:729] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.771785 5546 consensus_queue.cc:260] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "5cef4772b54342fe9d418f103c30fc33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44423 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } }
I20240503 07:59:17.772025 5546 ts_tablet_manager.cc:1398] T 1f0a00c4bec748bc957283814ca26ae1 P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.772164 5546 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Bootstrap starting.
I20240503 07:59:17.800717 5546 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.801097 5546 tablet_bootstrap.cc:492] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Bootstrap complete.
I20240503 07:59:17.802068 5546 ts_tablet_manager.cc:1367] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.009s
I20240503 07:59:17.802516 5546 raft_consensus.cc:348] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:59:17.802661 5546 raft_consensus.cc:729] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d51a644d914549b3dfb3f104d396e8, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.802816 5546 consensus_queue.cc:260] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "80bd2b9fafaf44bcb8c0f1b9f408841e" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39363 } } peers { permanent_uuid: "7a36d94f14d04d74b632bc75d1779bc3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38831 } } peers { permanent_uuid: "27d51a644d914549b3dfb3f104d396e8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36481 } attrs { replace: true } } peers { permanent_uuid: "b8f135eb22964d4a8d8300956bce8017" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35843 } attrs { promote: true } }
I20240503 07:59:17.803175 5546 ts_tablet_manager.cc:1398] T 1eca5e6693b24d0e8f3d9c37a62a1bca P 27d51a644d914549b3dfb3f104d396e8: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:17.820389 2123 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.821719 5588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.823966 2385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:17.827126 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.833314 693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:17.842247 1992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.853349 2253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:17.862679 812 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:17.891197 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 5522
W20240503 07:59:17.897306 780 connection.cc:535] client connection to 127.0.99.129:36481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 07:59:17.897470 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 632
W20240503 07:59:17.899178 780 consensus_peers.cc:485] T 97138ab5e3394c41b83382a22df0a198 P 5cef4772b54342fe9d418f103c30fc33 -> Peer 27d51a644d914549b3dfb3f104d396e8 (127.0.99.129:36481): Couldn't send request to peer 27d51a644d914549b3dfb3f104d396e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:36481: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 07:59:17.903797 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 763
I20240503 07:59:17.909638 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 894
I20240503 07:59:17.915418 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 1930
I20240503 07:59:17.920509 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 2061
I20240503 07:59:17.925355 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 2192
I20240503 07:59:17.930083 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 2323
I20240503 07:59:17.934963 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 410
2024-05-03T07:59:17Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (50623 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (50623 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 07:59:17.966444 398 test_util.cc:268] Using random seed: 414711397
I20240503 07:59:17.967227 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:17.967274 398 ts_itest-base.cc:113] --------------
I20240503 07:59:17.967284 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:59:17.967293 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:17.967301 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:17Z Disabled control of system clock
I20240503 07:59:17.974643 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36407
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36407
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 07:59:18.077637 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.077874 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:18.077930 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.080919 5761 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:18.080996 5761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:18.081022 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:18.081053 5761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:18.081084 5761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:18.084609 5761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36407
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:18.085448 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:18.086544 5761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:18.093688 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.094307 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.094792 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:18.095479 5761 server_base.cc:1034] running on GCE node
I20240503 07:59:18.095942 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:18.096792 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:18.097944 5761 hybrid_clock.cc:648] HybridClock initialized: now 1714723158097935 us; error 27 us; skew 500 ppm
I20240503 07:59:18.099726 5761 webserver.cc:460] Webserver started at http://127.0.99.190:44089/ using document root <none> and password file <none>
I20240503 07:59:18.100257 5761 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:18.100315 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:18.100610 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:18.102313 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.102802 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.106053 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:18.108121 5775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.109058 5761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.109161 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.109234 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:18.121093 5761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:18.121618 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:18.121757 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:18.127660 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36407 every 8 connection(s)
I20240503 07:59:18.127656 5761 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36407
I20240503 07:59:18.128574 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:18.128808 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 5761
I20240503 07:59:18.128895 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:18.131469 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.136267 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0: Bootstrap starting.
I20240503 07:59:18.138161 5828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.138808 5828 log.cc:824] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:18.140333 5828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0: No bootstrap required, opened a new log
I20240503 07:59:18.142660 5828 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } }
I20240503 07:59:18.142827 5828 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.142874 5828 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5a3aaee1ee472c9b00f3a0504b41b0, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.143244 5828 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } }
I20240503 07:59:18.143349 5828 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:18.143389 5828 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:18.143461 5828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.144184 5828 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } }
I20240503 07:59:18.144456 5828 leader_election.cc:302] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5a3aaee1ee472c9b00f3a0504b41b0; no voters:
I20240503 07:59:18.144670 5828 leader_election.cc:288] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:18.144747 5833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.145174 5833 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [term 1 LEADER]: Becoming Leader. State: Replica: ca5a3aaee1ee472c9b00f3a0504b41b0, State: Running, Role: LEADER
I20240503 07:59:18.145430 5833 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } }
I20240503 07:59:18.145681 5828 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:18.146703 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5a3aaee1ee472c9b00f3a0504b41b0. Latest consensus state: current_term: 1 leader_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } } }
I20240503 07:59:18.146826 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:18.147171 5842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:18.147127 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5a3aaee1ee472c9b00f3a0504b41b0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36407 } } }
I20240503 07:59:18.147598 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:18.150087 5842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:18.154723 5842 catalog_manager.cc:1347] Generated new cluster ID: 4c8e2b16d4ef4a1686723946abbc72de
I20240503 07:59:18.154791 5842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:18.178001 5842 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:18.178875 5842 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:18.201653 5842 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ca5a3aaee1ee472c9b00f3a0504b41b0: Generated new TSK 0
I20240503 07:59:18.202387 5842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:18.206827 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:18.328301 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.328490 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:18.328533 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.331444 5852 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:18.331506 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:18.331589 5852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:18.335032 5852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--never_fsync=true
--heap_profile_path=/tmp/kudu.5852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:18.335799 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:18.336855 5852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:18.344910 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.345525 5860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.346189 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:18.346698 5852 server_base.cc:1034] running on GCE node
I20240503 07:59:18.347165 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:18.347684 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:18.348853 5852 hybrid_clock.cc:648] HybridClock initialized: now 1714723158348839 us; error 30 us; skew 500 ppm
I20240503 07:59:18.350679 5852 webserver.cc:460] Webserver started at http://127.0.99.129:35479/ using document root <none> and password file <none>
I20240503 07:59:18.351244 5852 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:18.351300 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:18.351467 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:18.353099 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4fffa62732434082a3cba9c4458854a3"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.353542 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4fffa62732434082a3cba9c4458854a3"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.356961 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:18.359483 5866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.360584 5852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:18.360683 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4fffa62732434082a3cba9c4458854a3"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.360796 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:18.377002 5852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:18.377607 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:18.377761 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:18.378278 5852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:18.379158 5852 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:18.379204 5852 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.379242 5852 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:18.379256 5852 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.387218 5852 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36161
I20240503 07:59:18.387279 5979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36161 every 8 connection(s)
I20240503 07:59:18.388110 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:18.392602 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 5852
I20240503 07:59:18.392691 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:18.394488 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:18.396451 5980 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36407
I20240503 07:59:18.396664 5980 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:18.397147 5980 heartbeater.cc:511] Master 127.0.99.190:36407 requested a full tablet report, sending...
I20240503 07:59:18.398633 5792 ts_manager.cc:194] Registered new tserver with Master: 4fffa62732434082a3cba9c4458854a3 (127.0.99.129:36161)
I20240503 07:59:18.400252 5792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:45217
W20240503 07:59:18.497848 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.498055 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:18.498102 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.501084 5983 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:18.501148 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:18.501215 5983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:18.504707 5983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--never_fsync=true
--heap_profile_path=/tmp/kudu.5983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:18.505537 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:18.506575 5983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:18.515170 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.515815 5991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.516383 5989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:18.517038 5983 server_base.cc:1034] running on GCE node
I20240503 07:59:18.517422 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:18.517952 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:18.519098 5983 hybrid_clock.cc:648] HybridClock initialized: now 1714723158519084 us; error 35 us; skew 500 ppm
I20240503 07:59:18.520893 5983 webserver.cc:460] Webserver started at http://127.0.99.130:35405/ using document root <none> and password file <none>
I20240503 07:59:18.521404 5983 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:18.521453 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:18.521597 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:18.523200 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6285209a7d244bcdb5d772843fb7f45d"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.523607 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6285209a7d244bcdb5d772843fb7f45d"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.526623 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:18.528623 5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.529649 5983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:18.529739 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6285209a7d244bcdb5d772843fb7f45d"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.529819 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:18.546026 5983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:18.546617 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:18.546761 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:18.547323 5983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:18.548171 5983 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:18.548215 5983 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.548255 5983 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:18.548282 5983 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.556375 5983 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39217
I20240503 07:59:18.556401 6110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39217 every 8 connection(s)
I20240503 07:59:18.557250 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:18.559698 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 5983
I20240503 07:59:18.559799 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:18.561656 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:18.565665 6111 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36407
I20240503 07:59:18.565886 6111 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:18.566290 6111 heartbeater.cc:511] Master 127.0.99.190:36407 requested a full tablet report, sending...
I20240503 07:59:18.567278 5792 ts_manager.cc:194] Registered new tserver with Master: 6285209a7d244bcdb5d772843fb7f45d (127.0.99.130:39217)
I20240503 07:59:18.567999 5792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:34965
W20240503 07:59:18.663708 6114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.663951 6114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:18.663998 6114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.667065 6114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:18.667126 6114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:18.667191 6114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:18.670673 6114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--never_fsync=true
--heap_profile_path=/tmp/kudu.6114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:18.671526 6114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:18.672585 6114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:18.680764 6119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.681358 6120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.681946 6122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:18.682616 6114 server_base.cc:1034] running on GCE node
I20240503 07:59:18.683022 6114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:18.683538 6114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:18.684680 6114 hybrid_clock.cc:648] HybridClock initialized: now 1714723158684656 us; error 44 us; skew 500 ppm
I20240503 07:59:18.686497 6114 webserver.cc:460] Webserver started at http://127.0.99.131:38577/ using document root <none> and password file <none>
I20240503 07:59:18.687032 6114 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:18.687088 6114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:18.687230 6114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:18.688764 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "46db5909fbeb4f4e91f1a441a8f14649"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.689193 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "46db5909fbeb4f4e91f1a441a8f14649"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.692312 6114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:18.694388 6128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.695417 6114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:18.695513 6114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "46db5909fbeb4f4e91f1a441a8f14649"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.695583 6114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:18.708353 6114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:18.708894 6114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:18.709021 6114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:18.709471 6114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:18.710304 6114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:18.710347 6114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.710384 6114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:18.710403 6114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.718319 6114 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:33987
I20240503 07:59:18.718408 6241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:33987 every 8 connection(s)
I20240503 07:59:18.719229 6114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:18.726477 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6114
I20240503 07:59:18.726567 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:18.728386 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:18.731956 6242 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36407
I20240503 07:59:18.732161 6242 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:18.732589 6242 heartbeater.cc:511] Master 127.0.99.190:36407 requested a full tablet report, sending...
I20240503 07:59:18.733495 5792 ts_manager.cc:194] Registered new tserver with Master: 46db5909fbeb4f4e91f1a441a8f14649 (127.0.99.131:33987)
I20240503 07:59:18.734269 5792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:43939
W20240503 07:59:18.829316 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.829511 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:18.829551 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.832516 6245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:18.832573 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:18.832643 6245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:18.836129 6245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:18.836925 6245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:18.837942 6245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:18.845955 6253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.846547 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:18.847095 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:18.847689 6245 server_base.cc:1034] running on GCE node
I20240503 07:59:18.848040 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:18.848536 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:18.849684 6245 hybrid_clock.cc:648] HybridClock initialized: now 1714723158849668 us; error 35 us; skew 500 ppm
I20240503 07:59:18.851553 6245 webserver.cc:460] Webserver started at http://127.0.99.132:39515/ using document root <none> and password file <none>
I20240503 07:59:18.852077 6245 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:18.852128 6245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:18.852298 6245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:18.853933 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4ee16c7d6b06487ebf5f08a9edc3459c"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.854369 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4ee16c7d6b06487ebf5f08a9edc3459c"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.857517 6245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:18.859596 6259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.860580 6245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.860669 6245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4ee16c7d6b06487ebf5f08a9edc3459c"
format_stamp: "Formatted at 2024-05-03 07:59:18 on dist-test-slave-11df"
I20240503 07:59:18.860738 6245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:18.884601 6245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:18.885164 6245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:18.885293 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:18.885775 6245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:18.886600 6245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:18.886651 6245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.886698 6245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:18.886726 6245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:18.894752 6245 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45205
I20240503 07:59:18.894783 6372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45205 every 8 connection(s)
I20240503 07:59:18.895690 6245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:18.903431 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6245
I20240503 07:59:18.903539 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:18.904212 6373 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36407
I20240503 07:59:18.904435 6373 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:18.904943 6373 heartbeater.cc:511] Master 127.0.99.190:36407 requested a full tablet report, sending...
I20240503 07:59:18.905599 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--builtin_ntp_servers=127.0.99.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:18.905841 5792 ts_manager.cc:194] Registered new tserver with Master: 4ee16c7d6b06487ebf5f08a9edc3459c (127.0.99.132:45205)
I20240503 07:59:18.906366 5792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:42953
W20240503 07:59:19.008783 6376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:19.008980 6376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:19.009020 6376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:19.012027 6376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:19.012085 6376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:19.012168 6376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:19.015655 6376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36407
--never_fsync=true
--heap_profile_path=/tmp/kudu.6376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:19.016438 6376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:19.017438 6376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:19.025606 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:19.026154 6382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:19.026857 6381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:19.027526 6376 server_base.cc:1034] running on GCE node
I20240503 07:59:19.027901 6376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:19.028458 6376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:19.029598 6376 hybrid_clock.cc:648] HybridClock initialized: now 1714723159029577 us; error 42 us; skew 500 ppm
I20240503 07:59:19.031402 6376 webserver.cc:460] Webserver started at http://127.0.99.133:38503/ using document root <none> and password file <none>
I20240503 07:59:19.031917 6376 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:19.031973 6376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:19.032135 6376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:19.033805 6376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5f5f5054426443eb89256d7bea706f41"
format_stamp: "Formatted at 2024-05-03 07:59:19 on dist-test-slave-11df"
I20240503 07:59:19.034247 6376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5f5f5054426443eb89256d7bea706f41"
format_stamp: "Formatted at 2024-05-03 07:59:19 on dist-test-slave-11df"
I20240503 07:59:19.037324 6376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:19.039294 6390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:19.040210 6376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.040305 6376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5f5f5054426443eb89256d7bea706f41"
format_stamp: "Formatted at 2024-05-03 07:59:19 on dist-test-slave-11df"
I20240503 07:59:19.040381 6376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:19.056370 6376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:19.056943 6376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:19.057090 6376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:19.057590 6376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:19.058462 6376 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:19.058507 6376 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:19.058545 6376 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:19.058573 6376 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:19.066612 6376 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40881
I20240503 07:59:19.066658 6503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40881 every 8 connection(s)
I20240503 07:59:19.067533 6376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:19.070326 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6376
I20240503 07:59:19.070418 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:19.075515 6504 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36407
I20240503 07:59:19.075702 6504 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:19.076099 6504 heartbeater.cc:511] Master 127.0.99.190:36407 requested a full tablet report, sending...
I20240503 07:59:19.076908 5792 ts_manager.cc:194] Registered new tserver with Master: 5f5f5054426443eb89256d7bea706f41 (127.0.99.133:40881)
I20240503 07:59:19.077422 5792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:59569
I20240503 07:59:19.084065 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:19.095760 5792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42692:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:19.111685 6045 tablet_service.cc:1467] Processing CreateTablet for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f (DEFAULT_TABLE table=TestTable [id=8f8bb31085914b938cada4243e559ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.112031 6176 tablet_service.cc:1467] Processing CreateTablet for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f (DEFAULT_TABLE table=TestTable [id=8f8bb31085914b938cada4243e559ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.112677 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.112519 6307 tablet_service.cc:1467] Processing CreateTablet for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f (DEFAULT_TABLE table=TestTable [id=8f8bb31085914b938cada4243e559ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.112851 6176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.113245 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcecfc2ac5b4ebfb9288a14cfc8cf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.117493 6521 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Bootstrap starting.
I20240503 07:59:19.118573 6523 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Bootstrap starting.
I20240503 07:59:19.119952 6521 tablet_bootstrap.cc:654] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.120174 6523 tablet_bootstrap.cc:654] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.120635 6521 log.cc:824] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:19.120746 6523 log.cc:824] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:19.121280 6522 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Bootstrap starting.
I20240503 07:59:19.122049 6521 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: No bootstrap required, opened a new log
I20240503 07:59:19.122195 6521 ts_tablet_manager.cc:1367] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:19.122830 6522 tablet_bootstrap.cc:654] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.123432 6522 log.cc:824] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:19.124596 6523 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: No bootstrap required, opened a new log
I20240503 07:59:19.124743 6523 ts_tablet_manager.cc:1367] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 07:59:19.124904 6522 tablet_bootstrap.cc:492] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: No bootstrap required, opened a new log
I20240503 07:59:19.125061 6522 ts_tablet_manager.cc:1367] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:19.125473 6521 raft_consensus.cc:348] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.125691 6521 raft_consensus.cc:374] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.125751 6521 raft_consensus.cc:729] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee16c7d6b06487ebf5f08a9edc3459c, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.126176 6521 consensus_queue.cc:260] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.126840 6373 heartbeater.cc:503] Master 127.0.99.190:36407 was elected leader, sending a full tablet report...
I20240503 07:59:19.127034 6521 ts_tablet_manager.cc:1398] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:59:19.127861 6523 raft_consensus.cc:348] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.128021 6523 raft_consensus.cc:374] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.128067 6523 raft_consensus.cc:729] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46db5909fbeb4f4e91f1a441a8f14649, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.128399 6523 consensus_queue.cc:260] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.128918 6242 heartbeater.cc:503] Master 127.0.99.190:36407 was elected leader, sending a full tablet report...
I20240503 07:59:19.128844 6522 raft_consensus.cc:348] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.129026 6522 raft_consensus.cc:374] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.129072 6522 raft_consensus.cc:729] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6285209a7d244bcdb5d772843fb7f45d, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.129408 6522 consensus_queue.cc:260] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.129948 6523 ts_tablet_manager.cc:1398] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:19.130069 6522 ts_tablet_manager.cc:1398] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:19.130353 6111 heartbeater.cc:503] Master 127.0.99.190:36407 was elected leader, sending a full tablet report...
W20240503 07:59:19.146544 6374 tablet.cc:2369] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:19.220214 6243 tablet.cc:2369] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:19.308468 6112 tablet.cc:2369] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:19.402516 5980 heartbeater.cc:503] Master 127.0.99.190:36407 was elected leader, sending a full tablet report...
I20240503 07:59:19.478368 6528 raft_consensus.cc:482] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.478590 6528 raft_consensus.cc:504] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.479673 6528 leader_election.cc:288] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6285209a7d244bcdb5d772843fb7f45d (127.0.99.130:39217), 4ee16c7d6b06487ebf5f08a9edc3459c (127.0.99.132:45205)
I20240503 07:59:19.483179 6327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcecfc2ac5b4ebfb9288a14cfc8cf1f" candidate_uuid: "46db5909fbeb4f4e91f1a441a8f14649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" is_pre_election: true
I20240503 07:59:19.483186 6065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcecfc2ac5b4ebfb9288a14cfc8cf1f" candidate_uuid: "46db5909fbeb4f4e91f1a441a8f14649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285209a7d244bcdb5d772843fb7f45d" is_pre_election: true
I20240503 07:59:19.483484 6327 raft_consensus.cc:2454] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46db5909fbeb4f4e91f1a441a8f14649 in term 0.
I20240503 07:59:19.483482 6065 raft_consensus.cc:2454] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46db5909fbeb4f4e91f1a441a8f14649 in term 0.
I20240503 07:59:19.483928 6131 leader_election.cc:302] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46db5909fbeb4f4e91f1a441a8f14649, 4ee16c7d6b06487ebf5f08a9edc3459c; no voters:
I20240503 07:59:19.484165 6528 raft_consensus.cc:2781] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.484242 6528 raft_consensus.cc:482] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.484272 6528 raft_consensus.cc:3037] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.485103 6528 raft_consensus.cc:504] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.485435 6528 leader_election.cc:288] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [CANDIDATE]: Term 1 election: Requested vote from peers 6285209a7d244bcdb5d772843fb7f45d (127.0.99.130:39217), 4ee16c7d6b06487ebf5f08a9edc3459c (127.0.99.132:45205)
I20240503 07:59:19.485759 6065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcecfc2ac5b4ebfb9288a14cfc8cf1f" candidate_uuid: "46db5909fbeb4f4e91f1a441a8f14649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285209a7d244bcdb5d772843fb7f45d"
I20240503 07:59:19.485755 6327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcecfc2ac5b4ebfb9288a14cfc8cf1f" candidate_uuid: "46db5909fbeb4f4e91f1a441a8f14649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c"
I20240503 07:59:19.485873 6065 raft_consensus.cc:3037] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.485873 6327 raft_consensus.cc:3037] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.486877 6065 raft_consensus.cc:2454] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 6285209a7d244bcdb5d772843fb7f45d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46db5909fbeb4f4e91f1a441a8f14649 in term 1.
I20240503 07:59:19.486877 6327 raft_consensus.cc:2454] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 4ee16c7d6b06487ebf5f08a9edc3459c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46db5909fbeb4f4e91f1a441a8f14649 in term 1.
I20240503 07:59:19.487179 6131 leader_election.cc:302] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46db5909fbeb4f4e91f1a441a8f14649, 4ee16c7d6b06487ebf5f08a9edc3459c; no voters:
I20240503 07:59:19.487332 6528 raft_consensus.cc:2781] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.487540 6528 raft_consensus.cc:686] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [term 1 LEADER]: Becoming Leader. State: Replica: 46db5909fbeb4f4e91f1a441a8f14649, State: Running, Role: LEADER
I20240503 07:59:19.487848 6528 consensus_queue.cc:237] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } }
I20240503 07:59:19.490447 5792 catalog_manager.cc:5484] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46db5909fbeb4f4e91f1a441a8f14649 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "46db5909fbeb4f4e91f1a441a8f14649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 } } peers { permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 } } peers { permanent_uuid: "46db5909fbeb4f4e91f1a441a8f14649" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33987 } } }
I20240503 07:59:19.561488 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:19.563107 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4fffa62732434082a3cba9c4458854a3 to finish bootstrapping
I20240503 07:59:19.568864 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6285209a7d244bcdb5d772843fb7f45d to finish bootstrapping
I20240503 07:59:19.573120 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 46db5909fbeb4f4e91f1a441a8f14649 to finish bootstrapping
I20240503 07:59:19.577291 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4ee16c7d6b06487ebf5f08a9edc3459c to finish bootstrapping
I20240503 07:59:19.581413 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5f5f5054426443eb89256d7bea706f41 to finish bootstrapping
I20240503 07:59:19.759527 6438 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.760757 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:19.762465 5914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.762683 6045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.763710 6307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.851673 6438 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.851845 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:19.853646 6307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.855746 6045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.871274 5914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:19.907517 6528 consensus_queue.cc:1035] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6285209a7d244bcdb5d772843fb7f45d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.912519 6528 consensus_queue.cc:1035] T ffcecfc2ac5b4ebfb9288a14cfc8cf1f P 46db5909fbeb4f4e91f1a441a8f14649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee16c7d6b06487ebf5f08a9edc3459c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.068694 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:20.069176 6438 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.070046 6307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.077694 6045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.079106 6504 heartbeater.cc:503] Master 127.0.99.190:36407 was elected leader, sending a full tablet report...
I20240503 07:59:20.082509 5914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.159132 5914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.159497 6307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.160044 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:20.162701 6438 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.180719 6045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.267632 6045 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.267993 6307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.268481 6438 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.269021 6176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:20.270601 5914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:20.305418 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 5852
I20240503 07:59:20.310456 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 5983
I20240503 07:59:20.315156 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6114
I20240503 07:59:20.320272 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6245
I20240503 07:59:20.324646 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6376
I20240503 07:59:20.328846 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 5761
2024-05-03T07:59:20Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2383 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2383 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240503 07:59:20.350054 398 test_util.cc:268] Using random seed: 417095008
I20240503 07:59:20.350734 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:20.350775 398 ts_itest-base.cc:113] --------------
I20240503 07:59:20.350785 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 07:59:20.350793 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 07:59:20.350801 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:20Z Disabled control of system clock
I20240503 07:59:20.357779 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43653
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43653
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 07:59:20.460003 6716 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 07:59:20.460247 6716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:20.460291 6716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:20.460350 6716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:20.463608 6716 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:20.463686 6716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:20.463719 6716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:20.463750 6716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:20.463784 6716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:20.467514 6716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43653
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:20.468400 6716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:20.469502 6716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:20.477438 6724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.477807 6722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.478308 6721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:20.479107 6716 server_base.cc:1034] running on GCE node
I20240503 07:59:20.479650 6716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:20.480590 6716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:20.481745 6716 hybrid_clock.cc:648] HybridClock initialized: now 1714723160481722 us; error 39 us; skew 500 ppm
I20240503 07:59:20.483718 6716 webserver.cc:460] Webserver started at http://127.0.99.190:33599/ using document root <none> and password file <none>
I20240503 07:59:20.484251 6716 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:20.484310 6716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:20.484493 6716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:20.486161 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8d366789da3a463e80dd995b3ff2311f"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.486608 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8d366789da3a463e80dd995b3ff2311f"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.489794 6716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:20.491940 6730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.492908 6716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.493016 6716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8d366789da3a463e80dd995b3ff2311f"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.493116 6716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:20.505818 6716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:20.506448 6716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:20.506595 6716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:20.512820 6716 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43653
I20240503 07:59:20.512830 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43653 every 8 connection(s)
I20240503 07:59:20.513765 6716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:20.516252 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.521063 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f: Bootstrap starting.
I20240503 07:59:20.522207 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6716
I20240503 07:59:20.522321 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:20.523324 6783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.524047 6783 log.cc:824] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:20.525653 6783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f: No bootstrap required, opened a new log
I20240503 07:59:20.528050 6783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } }
I20240503 07:59:20.528225 6783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.528272 6783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d366789da3a463e80dd995b3ff2311f, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.528661 6783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } }
I20240503 07:59:20.528780 6783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:20.528839 6783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:20.528942 6783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.529745 6783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } }
I20240503 07:59:20.530072 6783 leader_election.cc:302] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d366789da3a463e80dd995b3ff2311f; no voters:
I20240503 07:59:20.530328 6783 leader_election.cc:288] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:20.530699 6788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.531143 6788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [term 1 LEADER]: Becoming Leader. State: Replica: 8d366789da3a463e80dd995b3ff2311f, State: Running, Role: LEADER
I20240503 07:59:20.531416 6788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } }
I20240503 07:59:20.531459 6783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:20.532895 6789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d366789da3a463e80dd995b3ff2311f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } } }
I20240503 07:59:20.532892 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d366789da3a463e80dd995b3ff2311f. Latest consensus state: current_term: 1 leader_uuid: "8d366789da3a463e80dd995b3ff2311f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d366789da3a463e80dd995b3ff2311f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43653 } } }
I20240503 07:59:20.533026 6789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:20.533026 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:20.533396 6802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:20.536072 6802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:20.540238 6802 catalog_manager.cc:1347] Generated new cluster ID: 447f0b2694dd4837b6fdd42997ade3da
I20240503 07:59:20.540300 6802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:20.560130 6802 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:20.560983 6802 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:20.565536 6802 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8d366789da3a463e80dd995b3ff2311f: Generated new TSK 0
I20240503 07:59:20.566507 6802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:20.576907 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:20.679576 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:20.679778 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:20.679823 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:20.682961 6807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:20.683040 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:20.683135 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:20.686723 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:20.687589 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:20.688642 6807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:20.696722 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.697301 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.697835 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:20.698467 6807 server_base.cc:1034] running on GCE node
I20240503 07:59:20.698830 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:20.699406 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:20.700559 6807 hybrid_clock.cc:648] HybridClock initialized: now 1714723160700548 us; error 26 us; skew 500 ppm
I20240503 07:59:20.702363 6807 webserver.cc:460] Webserver started at http://127.0.99.129:39567/ using document root <none> and password file <none>
I20240503 07:59:20.702877 6807 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:20.702956 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:20.703152 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:20.704768 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "625353768527407d8849faefd6527f2d"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.705466 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "625353768527407d8849faefd6527f2d"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.708703 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:20.710703 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.711637 6807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:20.711733 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "625353768527407d8849faefd6527f2d"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.711812 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:20.743891 6807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:20.744560 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:20.744714 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:20.745249 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:20.746143 6807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:20.746186 6807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.746237 6807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:20.746263 6807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.754101 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43605
I20240503 07:59:20.754153 6934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43605 every 8 connection(s)
I20240503 07:59:20.755015 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:20.762238 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6807
I20240503 07:59:20.762362 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:20.763247 6935 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:20.763468 6935 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:20.763950 6935 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:20.764305 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:20.765358 6747 ts_manager.cc:194] Registered new tserver with Master: 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:20.766891 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:54611
W20240503 07:59:20.865962 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:20.866158 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:20.866199 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:20.869108 6938 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:20.869163 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:20.869225 6938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:20.872663 6938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:20.873448 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:20.874542 6938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:20.882464 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.883074 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.883582 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:20.884210 6938 server_base.cc:1034] running on GCE node
I20240503 07:59:20.884543 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:20.885039 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:20.886193 6938 hybrid_clock.cc:648] HybridClock initialized: now 1714723160886175 us; error 35 us; skew 500 ppm
I20240503 07:59:20.887964 6938 webserver.cc:460] Webserver started at http://127.0.99.130:41635/ using document root <none> and password file <none>
I20240503 07:59:20.888471 6938 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:20.888521 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:20.888664 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:20.890244 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "76091450575f4076bd09550833d46f68"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.890655 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "76091450575f4076bd09550833d46f68"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.893780 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:20.895910 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.896878 6938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.896972 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "76091450575f4076bd09550833d46f68"
format_stamp: "Formatted at 2024-05-03 07:59:20 on dist-test-slave-11df"
I20240503 07:59:20.897051 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:20.915342 6938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:20.915999 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:20.916174 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:20.916708 6938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:20.917590 6938 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:20.917635 6938 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.917673 6938 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:20.917686 6938 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.925705 6938 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43735
I20240503 07:59:20.925737 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43735 every 8 connection(s)
I20240503 07:59:20.926645 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:20.929622 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 6938
I20240503 07:59:20.929735 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:20.931711 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:20.935520 7066 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:20.935753 7066 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:20.936203 7066 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:20.937129 6747 ts_manager.cc:194] Registered new tserver with Master: 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:20.937691 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:45919
W20240503 07:59:21.037308 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:21.037518 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:21.037575 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:21.040581 7069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:21.040650 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:21.040730 7069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:21.044314 7069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:21.045225 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:21.046412 7069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:21.054544 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.055089 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.055658 7075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:21.056226 7069 server_base.cc:1034] running on GCE node
I20240503 07:59:21.056612 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:21.057147 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:21.058301 7069 hybrid_clock.cc:648] HybridClock initialized: now 1714723161058275 us; error 44 us; skew 500 ppm
I20240503 07:59:21.060094 7069 webserver.cc:460] Webserver started at http://127.0.99.131:33657/ using document root <none> and password file <none>
I20240503 07:59:21.060652 7069 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:21.060707 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:21.060868 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:21.062463 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "79339180bc914f17a9f9f196b671f8ad"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.062887 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "79339180bc914f17a9f9f196b671f8ad"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.066001 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:21.068058 7083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.069010 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:21.069104 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "79339180bc914f17a9f9f196b671f8ad"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.069182 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:21.085362 7069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:21.085942 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:21.086076 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:21.086562 7069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:21.087445 7069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:21.087491 7069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.087540 7069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:21.087564 7069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.095937 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38681
I20240503 07:59:21.095963 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38681 every 8 connection(s)
I20240503 07:59:21.096810 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:21.097544 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 7069
I20240503 07:59:21.097628 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:21.099676 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:21.105322 7197 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:21.105518 7197 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:21.105927 7197 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:21.106814 6747 ts_manager.cc:194] Registered new tserver with Master: 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:21.107365 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50629
W20240503 07:59:21.202212 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:21.202430 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:21.202494 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:21.205564 7200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:21.205631 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:21.205713 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:21.209314 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.7200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:21.210206 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:21.211370 7200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:21.219568 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.220075 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.220682 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:21.221311 7200 server_base.cc:1034] running on GCE node
I20240503 07:59:21.221664 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:21.222174 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:21.223322 7200 hybrid_clock.cc:648] HybridClock initialized: now 1714723161223296 us; error 44 us; skew 500 ppm
I20240503 07:59:21.225104 7200 webserver.cc:460] Webserver started at http://127.0.99.132:42915/ using document root <none> and password file <none>
I20240503 07:59:21.225616 7200 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:21.225668 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:21.225821 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:21.227504 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "faca354acbbb49c89f3bbf9b3798eae6"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.227941 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "faca354acbbb49c89f3bbf9b3798eae6"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.231089 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:21.233242 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.234233 7200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:21.234330 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "faca354acbbb49c89f3bbf9b3798eae6"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.234407 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:21.252805 7200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:21.253509 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:21.253660 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:21.254189 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:21.255131 7200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:21.255179 7200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.255242 7200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:21.255265 7200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.263374 7200 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35347
I20240503 07:59:21.263393 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35347 every 8 connection(s)
I20240503 07:59:21.264264 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:21.264878 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 7200
I20240503 07:59:21.264977 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:21.267061 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:21.273775 7328 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:21.273981 7328 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:21.274405 7328 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:21.275362 6747 ts_manager.cc:194] Registered new tserver with Master: faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347)
I20240503 07:59:21.276114 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60065
W20240503 07:59:21.372335 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:21.372539 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:21.372583 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:21.375622 7331 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:21.375684 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:21.375751 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:21.379359 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.7331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:21.380206 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:21.381285 7331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:21.389590 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.390156 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.390626 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:21.391212 7331 server_base.cc:1034] running on GCE node
I20240503 07:59:21.391585 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:21.392120 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:21.393261 7331 hybrid_clock.cc:648] HybridClock initialized: now 1714723161393238 us; error 37 us; skew 500 ppm
I20240503 07:59:21.395114 7331 webserver.cc:460] Webserver started at http://127.0.99.133:40133/ using document root <none> and password file <none>
I20240503 07:59:21.395674 7331 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:21.395730 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:21.395901 7331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:21.397617 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6dc3aabcda15467498f2d0f01bbececf"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.398059 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6dc3aabcda15467498f2d0f01bbececf"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.401413 7331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 07:59:21.403625 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.404604 7331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:21.404697 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6dc3aabcda15467498f2d0f01bbececf"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.404783 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:21.420445 7331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:21.421037 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:21.421180 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:21.421689 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:21.422514 7331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:21.422557 7331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.422600 7331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:21.422626 7331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.430724 7331 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42133
I20240503 07:59:21.430754 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42133 every 8 connection(s)
I20240503 07:59:21.431664 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:21.432384 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 7331
I20240503 07:59:21.432519 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:21.434724 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:21.441057 7459 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:21.441269 7459 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:21.441705 7459 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:21.442608 6747 ts_manager.cc:194] Registered new tserver with Master: 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133)
I20240503 07:59:21.443354 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:58501
W20240503 07:59:21.538360 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:21.538594 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:21.538643 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:21.541595 7462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:21.541657 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:21.541723 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:21.545244 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:21.546109 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:21.547209 7462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:21.555224 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.555794 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.556304 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:21.556913 7462 server_base.cc:1034] running on GCE node
I20240503 07:59:21.557276 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:21.557822 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:21.558965 7462 hybrid_clock.cc:648] HybridClock initialized: now 1714723161558951 us; error 26 us; skew 500 ppm
I20240503 07:59:21.560739 7462 webserver.cc:460] Webserver started at http://127.0.99.134:35255/ using document root <none> and password file <none>
I20240503 07:59:21.561272 7462 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:21.561329 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:21.561494 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:21.563112 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ac6ca2734007406792d116f67b4fb000"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.563552 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ac6ca2734007406792d116f67b4fb000"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.566972 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 07:59:21.569064 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.570056 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:21.570154 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ac6ca2734007406792d116f67b4fb000"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.570230 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:21.587661 7462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:21.588268 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:21.588409 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:21.588908 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:21.589956 7462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:21.590004 7462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.590055 7462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:21.590085 7462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.597965 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34081
I20240503 07:59:21.597985 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34081 every 8 connection(s)
I20240503 07:59:21.598876 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:21.600077 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 7462
I20240503 07:59:21.600193 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:21.602434 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:21.608117 7590 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:21.608389 7590 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:21.608974 7590 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:21.610020 6747 ts_manager.cc:194] Registered new tserver with Master: ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081)
I20240503 07:59:21.610586 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49497
W20240503 07:59:21.707729 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:21.707942 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:21.708000 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:21.711006 7593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:21.711069 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:21.711155 7593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:21.714677 7593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:21.715520 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:21.716605 7593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:21.724586 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.725143 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:21.725666 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:21.726305 7593 server_base.cc:1034] running on GCE node
I20240503 07:59:21.726663 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:21.727257 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:21.728397 7593 hybrid_clock.cc:648] HybridClock initialized: now 1714723161728383 us; error 28 us; skew 500 ppm
I20240503 07:59:21.730135 7593 webserver.cc:460] Webserver started at http://127.0.99.135:40113/ using document root <none> and password file <none>
I20240503 07:59:21.730683 7593 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:21.730738 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:21.730906 7593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:21.732658 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5629c2d36c84404dbef6e9e66860cc8d"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.733110 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5629c2d36c84404dbef6e9e66860cc8d"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.736258 7593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 07:59:21.738328 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.739296 7593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:59:21.739401 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5629c2d36c84404dbef6e9e66860cc8d"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:21.739478 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:21.766163 7593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:21.766906 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:21.767151 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:21.767727 7593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:21.768604 7593 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:21.768651 7593 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.768702 7593 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:21.768724 7593 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:21.769486 6935 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:21.777058 7593 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35189
I20240503 07:59:21.777078 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35189 every 8 connection(s)
I20240503 07:59:21.777927 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:21.785950 7721 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:21.786159 7721 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:21.786590 7721 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:21.787585 6747 ts_manager.cc:194] Registered new tserver with Master: 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189)
I20240503 07:59:21.788123 6747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:35293
I20240503 07:59:21.788177 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 7593
I20240503 07:59:21.788287 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:59:21.790539 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:21.799293 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:21.816929 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 (DEFAULT_TABLE table=TestTable [id=2d867d749b594723b00b88b6cd052156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.816949 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 (DEFAULT_TABLE table=TestTable [id=2d867d749b594723b00b88b6cd052156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.817940 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.817942 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.823730 7737 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:21.824370 7738 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:21.826162 7737 tablet_bootstrap.cc:654] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.826162 7738 tablet_bootstrap.cc:654] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.826978 7738 log.cc:824] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:21.826978 7737 log.cc:824] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:21.828473 7737 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:21.828474 7738 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:21.828640 7737 ts_tablet_manager.cc:1367] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:21.828640 7738 ts_tablet_manager.cc:1367] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:21.831040 7738 raft_consensus.cc:348] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:21.831259 7738 raft_consensus.cc:374] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.831312 7738 raft_consensus.cc:729] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.831498 7737 raft_consensus.cc:348] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:21.831693 7737 raft_consensus.cc:374] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.831787 7737 raft_consensus.cc:729] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.831743 7738 consensus_queue.cc:260] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:21.832157 7737 consensus_queue.cc:260] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:21.832430 7066 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:21.832619 7738 ts_tablet_manager.cc:1398] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:21.832967 7737 ts_tablet_manager.cc:1398] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20240503 07:59:21.928037 7067 tablet.cc:2369] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:22.006651 6936 tablet.cc:2369] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:22.109129 7197 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:22.132074 7742 raft_consensus.cc:482] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.132263 7742 raft_consensus.cc:504] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:22.133318 7742 leader_election.cc:288] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:22.136858 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381a34bdbd84c9ca7dbe8d5aca087f2" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68" is_pre_election: true
I20240503 07:59:22.137135 7020 raft_consensus.cc:2454] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:22.137643 6823 leader_election.cc:302] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:22.137879 7742 raft_consensus.cc:2781] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.137951 7742 raft_consensus.cc:482] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.137996 7742 raft_consensus.cc:3037] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.138844 7742 raft_consensus.cc:504] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:22.139163 7742 leader_election.cc:288] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:22.139575 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8381a34bdbd84c9ca7dbe8d5aca087f2" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:22.139690 7020 raft_consensus.cc:3037] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.140475 7020 raft_consensus.cc:2454] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:22.140743 6823 leader_election.cc:302] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:22.140884 7742 raft_consensus.cc:2781] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.141084 7742 raft_consensus.cc:686] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:22.141397 7742 consensus_queue.cc:237] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:22.144114 6747 catalog_manager.cc:5484] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } }
I20240503 07:59:22.170573 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:22.171901 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 625353768527407d8849faefd6527f2d to finish bootstrapping
I20240503 07:59:22.177232 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 76091450575f4076bd09550833d46f68 to finish bootstrapping
I20240503 07:59:22.181607 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 79339180bc914f17a9f9f196b671f8ad to finish bootstrapping
I20240503 07:59:22.186995 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver faca354acbbb49c89f3bbf9b3798eae6 to finish bootstrapping
I20240503 07:59:22.191959 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6dc3aabcda15467498f2d0f01bbececf to finish bootstrapping
I20240503 07:59:22.197068 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ac6ca2734007406792d116f67b4fb000 to finish bootstrapping
I20240503 07:59:22.202134 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5629c2d36c84404dbef6e9e66860cc8d to finish bootstrapping
I20240503 07:59:22.207409 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 7200
I20240503 07:59:22.212052 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 7331
I20240503 07:59:22.218351 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 7462
I20240503 07:59:22.222397 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 7593
I20240503 07:59:22.584445 7742 consensus_queue.cc:1035] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.977599 6747 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:25.983995 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 73029d928463492abf2ba405ce28763c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.984333 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73029d928463492abf2ba405ce28763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.984431 6868 tablet_service.cc:1467] Processing CreateTablet for tablet f0088b17e63d431eaa49ef33a4af3b04 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.984581 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 7f94cbfb150547038338fbc0d892c8a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.984695 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0088b17e63d431eaa49ef33a4af3b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.984840 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f94cbfb150547038338fbc0d892c8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.987056 7765 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:25.987046 7764 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:25.987957 7764 tablet_bootstrap.cc:654] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.988001 7765 tablet_bootstrap.cc:654] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.989107 7765 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:25.989189 7765 ts_tablet_manager.cc:1367] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:25.989312 7764 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:25.989372 7764 ts_tablet_manager.cc:1367] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:25.989786 7764 raft_consensus.cc:348] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:25.989740 7765 raft_consensus.cc:348] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:25.989874 7765 raft_consensus.cc:374] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.989913 7765 raft_consensus.cc:729] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.989957 7764 raft_consensus.cc:374] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.990026 7764 raft_consensus.cc:729] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.990036 7765 consensus_queue.cc:260] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:25.990170 7764 consensus_queue.cc:260] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:25.990339 7765 ts_tablet_manager.cc:1398] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.990394 7764 ts_tablet_manager.cc:1398] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.990967 7764 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:25.991109 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 73029d928463492abf2ba405ce28763c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.991766 7764 tablet_bootstrap.cc:654] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.992172 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73029d928463492abf2ba405ce28763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.992945 7764 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:25.993013 7764 ts_tablet_manager.cc:1367] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:25.992178 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 7f94cbfb150547038338fbc0d892c8a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.993674 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f94cbfb150547038338fbc0d892c8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.994045 7764 raft_consensus.cc:348] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:25.994223 7764 raft_consensus.cc:374] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.994303 7764 raft_consensus.cc:729] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.994663 7764 consensus_queue.cc:260] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:25.991109 7129 tablet_service.cc:1467] Processing CreateTablet for tablet f0088b17e63d431eaa49ef33a4af3b04 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.995227 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0088b17e63d431eaa49ef33a4af3b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.995302 7764 ts_tablet_manager.cc:1398] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.998383 7770 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.000186 7770 tablet_bootstrap.cc:654] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.001036 7770 log.cc:824] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:26.002534 7770 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.002691 7770 ts_tablet_manager.cc:1367] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:26.005043 7770 raft_consensus.cc:348] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.005241 7770 raft_consensus.cc:374] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.005322 7770 raft_consensus.cc:729] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.005728 7770 consensus_queue.cc:260] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.006460 7770 ts_tablet_manager.cc:1398] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:26.006697 7770 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.007529 7770 tablet_bootstrap.cc:654] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.008498 7770 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.008585 7770 ts_tablet_manager.cc:1367] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.009035 7770 raft_consensus.cc:348] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.009213 7770 raft_consensus.cc:374] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.009277 7770 raft_consensus.cc:729] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.009415 7770 consensus_queue.cc:260] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.009651 7770 ts_tablet_manager.cc:1398] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.009871 7770 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.010632 7770 tablet_bootstrap.cc:654] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.011946 7770 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.012030 7770 ts_tablet_manager.cc:1367] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.012370 7770 raft_consensus.cc:348] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.012449 7770 raft_consensus.cc:374] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.012476 7770 raft_consensus.cc:729] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.012557 7770 consensus_queue.cc:260] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.012776 7770 ts_tablet_manager.cc:1398] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.018193 7768 raft_consensus.cc:482] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.018299 7768 raft_consensus.cc:504] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.019209 7768 leader_election.cc:288] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.022857 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:26.023161 7151 raft_consensus.cc:2454] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:26.023530 6954 leader_election.cc:302] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.023737 7768 raft_consensus.cc:2781] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.023793 7768 raft_consensus.cc:482] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.023834 7768 raft_consensus.cc:3037] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.024945 7768 raft_consensus.cc:504] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.025200 7768 leader_election.cc:288] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.025532 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:26.025625 7151 raft_consensus.cc:3037] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.026381 7151 raft_consensus.cc:2454] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:26.026717 6954 leader_election.cc:302] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.026890 7768 raft_consensus.cc:2781] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.027122 7768 raft_consensus.cc:686] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:26.027400 7768 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.029175 6746 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } }
W20240503 07:59:26.100955 7198 tablet.cc:2369] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:26.101367 7772 raft_consensus.cc:482] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.101476 7772 raft_consensus.cc:504] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.102486 7772 leader_election.cc:288] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:26.106397 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0088b17e63d431eaa49ef33a4af3b04" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d" is_pre_election: true
I20240503 07:59:26.106588 6889 raft_consensus.cc:2454] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 0.
I20240503 07:59:26.107015 7087 leader_election.cc:302] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.107241 7772 raft_consensus.cc:2781] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.107301 7772 raft_consensus.cc:482] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.107343 7772 raft_consensus.cc:3037] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.108117 7772 raft_consensus.cc:504] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.108366 7772 leader_election.cc:288] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:26.108752 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0088b17e63d431eaa49ef33a4af3b04" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:26.108860 6889 raft_consensus.cc:3037] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.109580 6889 raft_consensus.cc:2454] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 1.
I20240503 07:59:26.109825 7087 leader_election.cc:302] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.109972 7772 raft_consensus.cc:2781] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.110185 7772 raft_consensus.cc:686] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Leader. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:26.110482 7772 consensus_queue.cc:237] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.113150 6746 catalog_manager.cc:5484] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 79339180bc914f17a9f9f196b671f8ad (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
I20240503 07:59:26.129866 7757 raft_consensus.cc:482] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.129985 7757 raft_consensus.cc:504] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.130360 7757 leader_election.cc:288] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.133945 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73029d928463492abf2ba405ce28763c" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:26.134070 7151 raft_consensus.cc:2454] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:26.134315 6823 leader_election.cc:302] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.134471 7757 raft_consensus.cc:2781] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.134526 7757 raft_consensus.cc:482] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.134555 7757 raft_consensus.cc:3037] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.135357 7757 raft_consensus.cc:504] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.135604 7757 leader_election.cc:288] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.135942 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73029d928463492abf2ba405ce28763c" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:26.136036 7151 raft_consensus.cc:3037] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.136655 7151 raft_consensus.cc:2454] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:26.136902 6823 leader_election.cc:302] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.137044 7757 raft_consensus.cc:2781] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.137215 7757 raft_consensus.cc:686] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:26.137352 7757 consensus_queue.cc:237] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.138963 6746 catalog_manager.cc:5484] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
W20240503 07:59:26.244688 7785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:26.244936 7785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:26.247979 7785 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:26.343060 6889 raft_consensus.cc:1261] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.343659 7772 consensus_queue.cc:1035] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.345675 7151 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.346210 7768 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.350229 7150 raft_consensus.cc:1261] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.350548 7757 consensus_queue.cc:1035] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.352288 7818 mvcc.cc:205] Tried to move back new op lower bound from 7023506089343864832 to 7023506088050143232. Current Snapshot: MvccSnapshot[applied={T|T < 7023506089343864832}]
I20240503 07:59:26.354809 7822 mvcc.cc:205] Tried to move back new op lower bound from 7023506089343864832 to 7023506088050143232. Current Snapshot: MvccSnapshot[applied={T|T < 7023506089343864832}]
I20240503 07:59:26.364172 7820 mvcc.cc:205] Tried to move back new op lower bound from 7023506089366204416 to 7023506088499769344. Current Snapshot: MvccSnapshot[applied={T|T < 7023506089366204416}]
INSERT report
rows total: 2000
time total: 111.416 ms
time per row: 0.0557081 ms
I20240503 07:59:26.399449 6746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:26.404687 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 63ed07cc90364f058eb5ded00d3f7514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.404734 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 63ed07cc90364f058eb5ded00d3f7514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.405009 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ed07cc90364f058eb5ded00d3f7514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.405014 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ed07cc90364f058eb5ded00d3f7514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.405119 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfcbed75e424ea3a337b51ba0bdd9e4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.405376 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfcbed75e424ea3a337b51ba0bdd9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.406634 7764 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:26.407017 7765 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:26.407469 7764 tablet_bootstrap.cc:654] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.407774 7765 tablet_bootstrap.cc:654] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.407773 7770 tablet_bootstrap.cc:492] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.407816 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 746df8ea7adb4b8cbd2b1782762683e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.408012 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746df8ea7adb4b8cbd2b1782762683e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.408190 6999 tablet_service.cc:1467] Processing CreateTablet for tablet 746df8ea7adb4b8cbd2b1782762683e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.408396 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746df8ea7adb4b8cbd2b1782762683e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.408540 7770 tablet_bootstrap.cc:654] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.409409 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfcbed75e424ea3a337b51ba0bdd9e4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.409576 7770 tablet_bootstrap.cc:492] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.409615 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfcbed75e424ea3a337b51ba0bdd9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.409649 7770 ts_tablet_manager.cc:1367] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.409883 7764 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:26.409962 7764 ts_tablet_manager.cc:1367] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:26.410073 7770 raft_consensus.cc:348] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.410182 7770 raft_consensus.cc:374] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.410215 7770 raft_consensus.cc:729] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.410329 7770 consensus_queue.cc:260] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.410364 7764 raft_consensus.cc:348] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.410457 7764 raft_consensus.cc:374] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.410473 7765 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:26.410486 7764 raft_consensus.cc:729] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.410539 7765 ts_tablet_manager.cc:1367] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:26.410585 7770 ts_tablet_manager.cc:1398] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.410599 7764 consensus_queue.cc:260] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.410825 7764 ts_tablet_manager.cc:1398] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.410908 7765 raft_consensus.cc:348] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.411031 7765 raft_consensus.cc:374] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.411060 7765 raft_consensus.cc:729] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.411165 7765 consensus_queue.cc:260] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.411393 7765 ts_tablet_manager.cc:1398] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.411593 7765 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:26.412441 7765 tablet_bootstrap.cc:654] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.413722 7764 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:26.414568 7764 tablet_bootstrap.cc:654] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.415637 7765 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:26.415706 7765 ts_tablet_manager.cc:1367] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:26.415800 7764 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:26.415872 7764 ts_tablet_manager.cc:1367] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:26.416087 7765 raft_consensus.cc:348] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.416179 7765 raft_consensus.cc:374] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.416209 7765 raft_consensus.cc:729] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.416332 7765 consensus_queue.cc:260] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.416347 7764 raft_consensus.cc:348] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.416424 7764 raft_consensus.cc:374] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.416483 7764 raft_consensus.cc:729] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.416602 7764 consensus_queue.cc:260] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.416868 7764 ts_tablet_manager.cc:1398] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.417191 7765 ts_tablet_manager.cc:1398] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.417408 7765 tablet_bootstrap.cc:492] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:26.418238 7765 tablet_bootstrap.cc:654] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.419405 7765 tablet_bootstrap.cc:492] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:26.419474 7765 ts_tablet_manager.cc:1367] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:26.419764 7765 raft_consensus.cc:348] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.419838 7765 raft_consensus.cc:374] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.419857 7765 raft_consensus.cc:729] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.419950 7765 consensus_queue.cc:260] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.420253 7765 ts_tablet_manager.cc:1398] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.478488 7757 raft_consensus.cc:482] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.478629 7757 raft_consensus.cc:504] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.479053 7757 leader_election.cc:288] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:26.479534 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746df8ea7adb4b8cbd2b1782762683e2" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68" is_pre_election: true
I20240503 07:59:26.479679 7020 raft_consensus.cc:2454] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:26.480002 6823 leader_election.cc:302] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:26.480185 7757 raft_consensus.cc:2781] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.480244 7757 raft_consensus.cc:482] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.480270 7757 raft_consensus.cc:3037] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.481027 7757 raft_consensus.cc:504] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.481274 7757 leader_election.cc:288] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:26.481634 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746df8ea7adb4b8cbd2b1782762683e2" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:26.481742 7020 raft_consensus.cc:3037] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.482493 7020 raft_consensus.cc:2454] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:26.482775 6823 leader_election.cc:302] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:26.482967 7757 raft_consensus.cc:2781] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.483076 7757 raft_consensus.cc:686] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:26.483211 7757 consensus_queue.cc:237] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.484683 6744 catalog_manager.cc:5484] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } }
I20240503 07:59:26.660985 7768 raft_consensus.cc:482] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.661114 7768 raft_consensus.cc:504] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.661475 7768 leader_election.cc:288] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.662019 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfcbed75e424ea3a337b51ba0bdd9e4" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:26.662190 7150 raft_consensus.cc:2454] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:26.662606 6954 leader_election.cc:302] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.662850 7768 raft_consensus.cc:2781] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.662945 7768 raft_consensus.cc:482] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.662986 7768 raft_consensus.cc:3037] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.662961 7757 raft_consensus.cc:482] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.663057 7757 raft_consensus.cc:504] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.663357 7757 leader_election.cc:288] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:26.663725 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68" is_pre_election: true
I20240503 07:59:26.663828 7020 raft_consensus.cc:2454] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:26.663822 7768 raft_consensus.cc:504] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.664093 7768 leader_election.cc:288] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:26.664100 6823 leader_election.cc:302] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:26.664281 7757 raft_consensus.cc:2781] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.664338 7757 raft_consensus.cc:482] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.664376 7757 raft_consensus.cc:3037] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.664389 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfcbed75e424ea3a337b51ba0bdd9e4" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:26.664507 7150 raft_consensus.cc:3037] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.665068 7757 raft_consensus.cc:504] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.665256 7150 raft_consensus.cc:2454] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:26.665304 7757 leader_election.cc:288] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:26.665575 6954 leader_election.cc:302] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:26.665614 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:26.665692 7020 raft_consensus.cc:3037] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.665724 7768 raft_consensus.cc:2781] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.665809 7768 raft_consensus.cc:686] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:26.665967 7768 consensus_queue.cc:237] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.666410 7020 raft_consensus.cc:2454] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:26.666682 6823 leader_election.cc:302] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:26.666988 7757 raft_consensus.cc:2781] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.667071 7757 raft_consensus.cc:686] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:26.667182 7757 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.667474 6744 catalog_manager.cc:5484] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
I20240503 07:59:26.668393 6742 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } }
W20240503 07:59:26.811712 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:26.811949 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:26.814941 7845 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:26.911641 7150 raft_consensus.cc:1261] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.912194 7768 consensus_queue.cc:1035] T 0cfcbed75e424ea3a337b51ba0bdd9e4 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.913921 7020 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.913931 7019 raft_consensus.cc:1261] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.914626 7783 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.914907 7783 consensus_queue.cc:1035] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.409 ms
time per row: 0.0527044 ms
I20240503 07:59:26.959367 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:26.964951 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 61da585c42664c72b8624641b1dac5dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.964759 7130 tablet_service.cc:1467] Processing CreateTablet for tablet a37812b3d0a346e3a1bc9c57956f2ca5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.964951 6869 tablet_service.cc:1467] Processing CreateTablet for tablet a37812b3d0a346e3a1bc9c57956f2ca5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.965307 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37812b3d0a346e3a1bc9c57956f2ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.965487 7131 tablet_service.cc:1467] Processing CreateTablet for tablet 61da585c42664c72b8624641b1dac5dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.965924 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da585c42664c72b8624641b1dac5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.966089 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37812b3d0a346e3a1bc9c57956f2ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.967126 7895 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:26.967401 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da585c42664c72b8624641b1dac5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.967998 7895 tablet_bootstrap.cc:654] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.969772 7897 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.970507 7897 tablet_bootstrap.cc:654] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.971369 7897 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.971444 7897 ts_tablet_manager.cc:1367] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.971823 7897 raft_consensus.cc:348] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.971909 7897 raft_consensus.cc:374] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.971938 7897 raft_consensus.cc:729] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.972059 7897 consensus_queue.cc:260] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.972292 7897 ts_tablet_manager.cc:1398] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.972447 7897 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.973616 7897 tablet_bootstrap.cc:654] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.974608 7897 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.974682 7897 ts_tablet_manager.cc:1367] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:26.975075 7897 raft_consensus.cc:348] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.975165 7897 raft_consensus.cc:374] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.975194 7897 raft_consensus.cc:729] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.975224 7129 tablet_service.cc:1467] Processing CreateTablet for tablet a786879136e44b8d99311bec38c2f85d (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.975312 7897 consensus_queue.cc:260] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.975451 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a786879136e44b8d99311bec38c2f85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.975630 7897 ts_tablet_manager.cc:1398] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.977331 7897 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:26.977797 7000 tablet_service.cc:1467] Processing CreateTablet for tablet a786879136e44b8d99311bec38c2f85d (DEFAULT_TABLE table=rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.978075 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a786879136e44b8d99311bec38c2f85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.978166 7897 tablet_bootstrap.cc:654] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.979944 7902 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:26.980324 7895 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:26.980433 7895 ts_tablet_manager.cc:1367] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:26.980830 7902 tablet_bootstrap.cc:654] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.980903 7895 raft_consensus.cc:348] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.981015 7895 raft_consensus.cc:374] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.981045 7895 raft_consensus.cc:729] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.981170 7895 consensus_queue.cc:260] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:26.981418 7895 ts_tablet_manager.cc:1398] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.981588 7895 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:26.982064 7897 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:26.982199 7897 ts_tablet_manager.cc:1367] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:26.982414 7895 tablet_bootstrap.cc:654] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.983618 7897 raft_consensus.cc:348] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.983691 7895 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:26.983747 7897 raft_consensus.cc:374] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.983788 7897 raft_consensus.cc:729] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.983944 7895 ts_tablet_manager.cc:1367] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:26.983915 7897 consensus_queue.cc:260] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.984424 7895 raft_consensus.cc:348] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.984570 7895 raft_consensus.cc:374] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.984637 7897 ts_tablet_manager.cc:1398] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.984644 7895 raft_consensus.cc:729] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.984839 7895 consensus_queue.cc:260] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:26.985155 7902 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:26.985217 7895 ts_tablet_manager.cc:1398] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.985292 7902 ts_tablet_manager.cc:1367] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:26.985770 7902 raft_consensus.cc:348] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.985903 7902 raft_consensus.cc:374] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.985967 7902 raft_consensus.cc:729] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.986090 7902 consensus_queue.cc:260] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:26.986361 7902 ts_tablet_manager.cc:1398] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.106904 7900 raft_consensus.cc:482] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.107067 7900 raft_consensus.cc:504] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.107414 7900 leader_election.cc:288] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:27.107873 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d" is_pre_election: true
I20240503 07:59:27.108006 6889 raft_consensus.cc:2454] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 0.
I20240503 07:59:27.108330 7087 leader_election.cc:302] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.108505 7900 raft_consensus.cc:2781] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.108606 7900 raft_consensus.cc:482] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.108649 7900 raft_consensus.cc:3037] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.109740 7900 raft_consensus.cc:504] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.110060 7900 leader_election.cc:288] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:27.110440 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:27.110550 6889 raft_consensus.cc:3037] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.111390 6889 raft_consensus.cc:2454] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 1.
I20240503 07:59:27.111667 7087 leader_election.cc:302] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.111856 7900 raft_consensus.cc:2781] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.111929 7900 raft_consensus.cc:686] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Leader. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:27.112058 7900 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.113651 6739 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 79339180bc914f17a9f9f196b671f8ad (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } }
I20240503 07:59:27.114853 7757 raft_consensus.cc:482] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.114962 7757 raft_consensus.cc:504] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.115226 7757 leader_election.cc:288] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.115574 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61da585c42664c72b8624641b1dac5dd" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:27.115695 7150 raft_consensus.cc:2454] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:27.116010 6823 leader_election.cc:302] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.116151 7757 raft_consensus.cc:2781] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.116206 7757 raft_consensus.cc:482] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.116248 7757 raft_consensus.cc:3037] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.116894 7757 raft_consensus.cc:504] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.117133 7757 leader_election.cc:288] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.117439 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61da585c42664c72b8624641b1dac5dd" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:27.117573 7150 raft_consensus.cc:3037] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.118424 7150 raft_consensus.cc:2454] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:27.118675 6823 leader_election.cc:302] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.118817 7757 raft_consensus.cc:2781] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.118898 7757 raft_consensus.cc:686] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:27.119076 7757 consensus_queue.cc:237] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.120520 6739 catalog_manager.cc:5484] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
I20240503 07:59:27.310250 7768 raft_consensus.cc:482] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.310386 7768 raft_consensus.cc:504] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:27.310755 7768 leader_election.cc:288] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.311287 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a786879136e44b8d99311bec38c2f85d" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:27.311426 7150 raft_consensus.cc:2454] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:27.311723 6954 leader_election.cc:302] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.311899 7768 raft_consensus.cc:2781] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.311954 7768 raft_consensus.cc:482] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.311995 7768 raft_consensus.cc:3037] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.312908 7768 raft_consensus.cc:504] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:27.313158 7768 leader_election.cc:288] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.313546 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a786879136e44b8d99311bec38c2f85d" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:27.313647 7150 raft_consensus.cc:3037] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.314316 7150 raft_consensus.cc:2454] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:27.314581 6954 leader_election.cc:302] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.314733 7768 raft_consensus.cc:2781] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.314821 7768 raft_consensus.cc:686] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:27.314978 7768 consensus_queue.cc:237] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:27.316483 6739 catalog_manager.cc:5484] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } }
W20240503 07:59:27.449383 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:27.449613 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:27.452687 7908 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:27.529932 7150 raft_consensus.cc:1261] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.530959 6889 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.531235 7768 consensus_queue.cc:1035] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.531755 7900 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.537500 7151 raft_consensus.cc:1261] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.537917 7757 consensus_queue.cc:1035] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 83.004 ms
time per row: 0.041502 ms
I20240503 07:59:27.574970 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:27.579887 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 829a4c6cc5d4441d907509c203635672 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.580096 6999 tablet_service.cc:1467] Processing CreateTablet for tablet dfba18f2f9fe4d259a369ad071f8c274 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.580195 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829a4c6cc5d4441d907509c203635672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.580165 6868 tablet_service.cc:1467] Processing CreateTablet for tablet 829a4c6cc5d4441d907509c203635672 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.580350 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfba18f2f9fe4d259a369ad071f8c274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.580411 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829a4c6cc5d4441d907509c203635672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.582314 7959 tablet_bootstrap.cc:492] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:27.582198 6998 tablet_service.cc:1467] Processing CreateTablet for tablet e21fe5b5f5a04168850a773226658450 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.582489 6869 tablet_service.cc:1467] Processing CreateTablet for tablet e21fe5b5f5a04168850a773226658450 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.582561 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21fe5b5f5a04168850a773226658450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.582681 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21fe5b5f5a04168850a773226658450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.583117 7959 tablet_bootstrap.cc:654] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.584192 7959 tablet_bootstrap.cc:492] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:27.584282 7959 ts_tablet_manager.cc:1367] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.584662 7959 raft_consensus.cc:348] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.584750 7959 raft_consensus.cc:374] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.584779 7959 raft_consensus.cc:729] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.584745 7129 tablet_service.cc:1467] Processing CreateTablet for tablet dfba18f2f9fe4d259a369ad071f8c274 (DEFAULT_TABLE table=rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.584885 7959 consensus_queue.cc:260] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.585012 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfba18f2f9fe4d259a369ad071f8c274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.585088 7960 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:27.585134 7959 ts_tablet_manager.cc:1398] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:27.585260 7959 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:27.585813 7960 tablet_bootstrap.cc:654] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.586027 7959 tablet_bootstrap.cc:654] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.587044 7959 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:27.587074 7960 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:27.587123 7959 ts_tablet_manager.cc:1367] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.587146 7960 ts_tablet_manager.cc:1367] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.587466 7959 raft_consensus.cc:348] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.587502 7960 raft_consensus.cc:348] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.587543 7959 raft_consensus.cc:374] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.587571 7960 raft_consensus.cc:374] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.587570 7959 raft_consensus.cc:729] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.587611 7960 raft_consensus.cc:729] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.587687 7959 consensus_queue.cc:260] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.587733 7960 consensus_queue.cc:260] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.587917 7959 ts_tablet_manager.cc:1398] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:27.587980 7960 ts_tablet_manager.cc:1398] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.588088 7959 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:27.588166 7960 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:27.588841 7959 tablet_bootstrap.cc:654] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.588864 7960 tablet_bootstrap.cc:654] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.590027 7960 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:27.590122 7960 ts_tablet_manager.cc:1367] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:27.590502 7960 raft_consensus.cc:348] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.590584 7960 raft_consensus.cc:374] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.590610 7960 raft_consensus.cc:729] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.590623 7959 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:27.590693 7959 ts_tablet_manager.cc:1367] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.590716 7960 consensus_queue.cc:260] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.590963 7960 ts_tablet_manager.cc:1398] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.591172 7959 raft_consensus.cc:348] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.591235 7964 tablet_bootstrap.cc:492] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:27.591264 7959 raft_consensus.cc:374] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.591298 7959 raft_consensus.cc:729] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.591828 7959 consensus_queue.cc:260] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.591997 7964 tablet_bootstrap.cc:654] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.592124 7959 ts_tablet_manager.cc:1398] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.593261 7964 tablet_bootstrap.cc:492] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:27.593343 7964 ts_tablet_manager.cc:1367] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.593719 7964 raft_consensus.cc:348] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.593803 7964 raft_consensus.cc:374] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.593832 7964 raft_consensus.cc:729] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.593947 7964 consensus_queue.cc:260] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.594173 7964 ts_tablet_manager.cc:1398] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:27.595484 7757 raft_consensus.cc:482] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.595557 7757 raft_consensus.cc:504] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.595827 7757 leader_election.cc:288] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:27.596146 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829a4c6cc5d4441d907509c203635672" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68" is_pre_election: true
I20240503 07:59:27.596282 7020 raft_consensus.cc:2454] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 0.
I20240503 07:59:27.596554 6823 leader_election.cc:302] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:27.596693 7757 raft_consensus.cc:2781] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.596750 7757 raft_consensus.cc:482] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.596776 7757 raft_consensus.cc:3037] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.597509 7757 raft_consensus.cc:504] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.597733 7757 leader_election.cc:288] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:27.598001 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829a4c6cc5d4441d907509c203635672" candidate_uuid: "625353768527407d8849faefd6527f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:27.598089 7020 raft_consensus.cc:3037] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.598747 7020 raft_consensus.cc:2454] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625353768527407d8849faefd6527f2d in term 1.
I20240503 07:59:27.599004 6823 leader_election.cc:302] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:27.599141 7757 raft_consensus.cc:2781] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.599218 7757 raft_consensus.cc:686] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Leader. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:27.599332 7757 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.600692 6741 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 625353768527407d8849faefd6527f2d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } }
I20240503 07:59:27.621439 7768 raft_consensus.cc:482] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.621560 7768 raft_consensus.cc:504] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.621873 7768 leader_election.cc:288] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.622321 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba18f2f9fe4d259a369ad071f8c274" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:27.622464 7151 raft_consensus.cc:2454] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:27.622716 6954 leader_election.cc:302] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.622895 7768 raft_consensus.cc:2781] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.622987 7768 raft_consensus.cc:482] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.623028 7768 raft_consensus.cc:3037] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.623811 7768 raft_consensus.cc:504] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.624071 7768 leader_election.cc:288] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:27.624405 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfba18f2f9fe4d259a369ad071f8c274" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:27.624523 7151 raft_consensus.cc:3037] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.625339 7151 raft_consensus.cc:2454] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:27.625604 6954 leader_election.cc:302] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:27.625767 7768 raft_consensus.cc:2781] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.625860 7768 raft_consensus.cc:686] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:27.625986 7768 consensus_queue.cc:237] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:27.627471 6741 catalog_manager.cc:5484] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
I20240503 07:59:27.696067 7768 raft_consensus.cc:482] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.696192 7768 raft_consensus.cc:504] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.696630 7768 leader_election.cc:288] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:27.700258 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21fe5b5f5a04168850a773226658450" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d" is_pre_election: true
I20240503 07:59:27.700405 6889 raft_consensus.cc:2454] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:27.701006 6956 leader_election.cc:302] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:27.701211 7768 raft_consensus.cc:2781] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.701287 7768 raft_consensus.cc:482] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.701316 7768 raft_consensus.cc:3037] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.702107 7768 raft_consensus.cc:504] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.702373 7768 leader_election.cc:288] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:27.702721 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21fe5b5f5a04168850a773226658450" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:27.702818 6889 raft_consensus.cc:3037] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.703538 6889 raft_consensus.cc:2454] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:27.703783 6956 leader_election.cc:302] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:27.703927 7768 raft_consensus.cc:2781] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.704023 7768 raft_consensus.cc:686] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:27.704182 7768 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:27.705727 6741 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } }
W20240503 07:59:27.847280 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:27.847590 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:27.850935 7972 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:27.940757 7020 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.941300 7783 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.950975 6889 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.953464 7768 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.961830 7151 raft_consensus.cc:1261] T dfba18f2f9fe4d259a369ad071f8c274 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.962337 7776 consensus_queue.cc:1035] T dfba18f2f9fe4d259a369ad071f8c274 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.337 ms
time per row: 0.0506686 ms
I20240503 07:59:27.991535 6742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54344:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:27.997366 7129 tablet_service.cc:1467] Processing CreateTablet for tablet ba2f6cdcc0704518a2931544ab969df8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.997509 6869 tablet_service.cc:1467] Processing CreateTablet for tablet ba2f6cdcc0704518a2931544ab969df8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.997701 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2f6cdcc0704518a2931544ab969df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.997789 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2f6cdcc0704518a2931544ab969df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.998009 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2f2ab1fe39408e8ee235acee496da7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.998257 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2f2ab1fe39408e8ee235acee496da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.999224 6868 tablet_service.cc:1467] Processing CreateTablet for tablet ce487b8bb9694381923b19d4e48257c8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.999431 6868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce487b8bb9694381923b19d4e48257c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.999547 7964 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:28.000180 7959 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: Bootstrap starting.
I20240503 07:59:28.000731 7964 tablet_bootstrap.cc:654] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.000990 7959 tablet_bootstrap.cc:654] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.001364 7960 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:28.002378 7960 tablet_bootstrap.cc:654] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.002676 7131 tablet_service.cc:1467] Processing CreateTablet for tablet ce487b8bb9694381923b19d4e48257c8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.002761 7959 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: No bootstrap required, opened a new log
I20240503 07:59:28.002828 7959 ts_tablet_manager.cc:1367] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:28.002955 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce487b8bb9694381923b19d4e48257c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.003245 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2f2ab1fe39408e8ee235acee496da7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.003340 7960 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:28.003300 7959 raft_consensus.cc:348] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.003396 7959 raft_consensus.cc:374] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.003407 7960 ts_tablet_manager.cc:1367] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.003422 7959 raft_consensus.cc:729] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.003463 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2f2ab1fe39408e8ee235acee496da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.003530 7959 consensus_queue.cc:260] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.003764 7959 ts_tablet_manager.cc:1398] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:28.003816 7960 raft_consensus.cc:348] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.003899 7960 raft_consensus.cc:374] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.003926 7960 raft_consensus.cc:729] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.004035 7960 consensus_queue.cc:260] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.004297 7960 ts_tablet_manager.cc:1398] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.004448 7960 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Bootstrap starting.
I20240503 07:59:28.005270 7960 tablet_bootstrap.cc:654] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.006521 7964 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:28.006618 7964 ts_tablet_manager.cc:1367] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:28.006904 7960 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: No bootstrap required, opened a new log
I20240503 07:59:28.006997 7960 ts_tablet_manager.cc:1367] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:28.007107 7964 raft_consensus.cc:348] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.007195 7964 raft_consensus.cc:374] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.007222 7964 raft_consensus.cc:729] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.007334 7964 consensus_queue.cc:260] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.007411 7960 raft_consensus.cc:348] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.007484 7960 raft_consensus.cc:374] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.007510 7960 raft_consensus.cc:729] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.007594 7964 ts_tablet_manager.cc:1398] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.007614 7960 consensus_queue.cc:260] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.007746 7964 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:28.007851 7960 ts_tablet_manager.cc:1398] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.008617 7964 tablet_bootstrap.cc:654] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.009964 7964 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:28.010078 7964 ts_tablet_manager.cc:1367] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.010432 7964 raft_consensus.cc:348] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.010516 7964 raft_consensus.cc:374] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.010536 7964 raft_consensus.cc:729] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.010627 7964 consensus_queue.cc:260] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.010850 7964 ts_tablet_manager.cc:1398] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.011031 7964 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Bootstrap starting.
I20240503 07:59:28.011775 7964 tablet_bootstrap.cc:654] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.013624 7964 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: No bootstrap required, opened a new log
I20240503 07:59:28.013700 7964 ts_tablet_manager.cc:1367] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:28.014016 7964 raft_consensus.cc:348] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.014092 7964 raft_consensus.cc:374] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.014128 7964 raft_consensus.cc:729] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.014235 7964 consensus_queue.cc:260] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.014483 7964 ts_tablet_manager.cc:1398] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.052413 7772 raft_consensus.cc:482] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.052536 7772 raft_consensus.cc:504] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.052898 7772 leader_election.cc:288] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:28.053285 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2f6cdcc0704518a2931544ab969df8" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d" is_pre_election: true
I20240503 07:59:28.053416 6889 raft_consensus.cc:2454] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 0.
I20240503 07:59:28.053689 7087 leader_election.cc:302] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.053907 7772 raft_consensus.cc:2781] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.053974 7772 raft_consensus.cc:482] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.054003 7772 raft_consensus.cc:3037] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.054754 7772 raft_consensus.cc:504] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.055025 7772 leader_election.cc:288] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:28.055404 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2f6cdcc0704518a2931544ab969df8" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:28.055536 6889 raft_consensus.cc:3037] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.056269 6889 raft_consensus.cc:2454] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 1.
I20240503 07:59:28.056517 7087 leader_election.cc:302] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.056660 7772 raft_consensus.cc:2781] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.056792 7772 raft_consensus.cc:686] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Leader. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:28.056917 7772 consensus_queue.cc:237] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } }
I20240503 07:59:28.058305 6742 catalog_manager.cc:5484] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 79339180bc914f17a9f9f196b671f8ad (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } }
I20240503 07:59:28.138474 7772 raft_consensus.cc:482] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.138610 7772 raft_consensus.cc:504] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.138796 7776 raft_consensus.cc:482] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.138890 7776 raft_consensus.cc:504] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.138978 7772 leader_election.cc:288] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:28.139201 7776 leader_election.cc:288] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:28.139338 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d" is_pre_election: true
I20240503 07:59:28.139483 6889 raft_consensus.cc:2454] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 0.
I20240503 07:59:28.139544 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2f2ab1fe39408e8ee235acee496da7" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad" is_pre_election: true
I20240503 07:59:28.139659 7151 raft_consensus.cc:2454] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 0.
I20240503 07:59:28.139802 7087 leader_election.cc:302] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.139943 6954 leader_election.cc:302] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.140117 7776 raft_consensus.cc:2781] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.140173 7776 raft_consensus.cc:482] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.140168 7772 raft_consensus.cc:2781] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.140218 7772 raft_consensus.cc:482] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.140220 7776 raft_consensus.cc:3037] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.140261 7772 raft_consensus.cc:3037] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.140970 7772 raft_consensus.cc:504] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.140996 7776 raft_consensus.cc:504] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.141208 7772 leader_election.cc:288] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:28.141237 7776 leader_election.cc:288] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:28.141522 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8" candidate_uuid: "79339180bc914f17a9f9f196b671f8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:28.141516 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2f2ab1fe39408e8ee235acee496da7" candidate_uuid: "76091450575f4076bd09550833d46f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:28.141621 7151 raft_consensus.cc:3037] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.141621 6889 raft_consensus.cc:3037] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.142438 7151 raft_consensus.cc:2454] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76091450575f4076bd09550833d46f68 in term 1.
I20240503 07:59:28.142439 6889 raft_consensus.cc:2454] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79339180bc914f17a9f9f196b671f8ad in term 1.
I20240503 07:59:28.142704 6954 leader_election.cc:302] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.142843 7776 raft_consensus.cc:2781] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.142944 7776 raft_consensus.cc:686] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Leader. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:28.142946 7087 leader_election.cc:302] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:28.143072 7772 raft_consensus.cc:2781] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.143105 7776 consensus_queue.cc:237] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } }
I20240503 07:59:28.143337 7772 raft_consensus.cc:686] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Leader. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:28.143471 7772 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } }
I20240503 07:59:28.144677 6742 catalog_manager.cc:5484] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76091450575f4076bd09550833d46f68 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } }
I20240503 07:59:28.144773 6743 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 79339180bc914f17a9f9f196b671f8ad (127.0.99.131). New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } }
W20240503 07:59:28.258464 8015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:28.258709 8015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:28.261701 8015 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:28.338050 6889 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.338241 7151 raft_consensus.cc:1261] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.338495 7772 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.338608 7776 consensus_queue.cc:1035] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.341032 6888 raft_consensus.cc:1261] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.343901 7772 consensus_queue.cc:1035] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.354828 8043 mvcc.cc:205] Tried to move back new op lower bound from 7023506097509359616 to 7023506096715042816. Current Snapshot: MvccSnapshot[applied={T|T < 7023506097509359616}]
INSERT report
rows total: 2000
time total: 79.1955 ms
time per row: 0.0395978 ms
I20240503 07:59:28.380074 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:35347
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:43653
--webserver_port=42915
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:28.485980 8050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:28.486187 8050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:28.486233 8050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:28.489357 8050 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:28.489418 8050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:28.489500 8050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:28.493084 8050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:35347
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=42915
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.8050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:28.493917 8050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:28.495031 8050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:28.503546 8058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.504061 8055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.504727 8056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:28.505380 8050 server_base.cc:1034] running on GCE node
I20240503 07:59:28.505712 8050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:28.506244 8050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:28.507387 8050 hybrid_clock.cc:648] HybridClock initialized: now 1714723168507373 us; error 26 us; skew 500 ppm
I20240503 07:59:28.509197 8050 webserver.cc:460] Webserver started at http://127.0.99.132:42915/ using document root <none> and password file <none>
I20240503 07:59:28.509716 8050 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:28.509783 8050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:28.512972 8050 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:28.514847 8064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.515945 8050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.516075 8050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "faca354acbbb49c89f3bbf9b3798eae6"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:28.516480 8050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:28.527070 8050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:28.527630 8050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:28.527776 8050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:28.528276 8050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:28.529129 8050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:28.529181 8050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.529232 8050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:28.529256 8050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.537743 8050 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35347
I20240503 07:59:28.537765 8177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35347 every 8 connection(s)
I20240503 07:59:28.538712 8050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:28.544664 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 8050
I20240503 07:59:28.545024 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:42133
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:43653
--webserver_port=40133
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:28.547613 8178 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:28.547842 8178 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:28.548379 8178 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:28.549486 6740 ts_manager.cc:194] Re-registered known tserver with Master: faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347)
I20240503 07:59:28.550047 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45483
W20240503 07:59:28.649109 8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:28.649307 8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:28.649355 8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:28.652400 8181 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:28.652457 8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:28.652539 8181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:28.656085 8181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:42133
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40133
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.8181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:28.656916 8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:28.657997 8181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:28.666031 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.666504 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.667096 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:28.667657 8181 server_base.cc:1034] running on GCE node
I20240503 07:59:28.667980 8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:28.668496 8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:28.669634 8181 hybrid_clock.cc:648] HybridClock initialized: now 1714723168669623 us; error 28 us; skew 500 ppm
I20240503 07:59:28.671474 8181 webserver.cc:460] Webserver started at http://127.0.99.133:40133/ using document root <none> and password file <none>
I20240503 07:59:28.672035 8181 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:28.672103 8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:28.675307 8181 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:28.677078 8195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.678042 8181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.678138 8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6dc3aabcda15467498f2d0f01bbececf"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:28.678551 8181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:28.708910 8181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:28.709553 8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:28.709695 8181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:28.710213 8181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:28.711081 8181 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:28.711128 8181 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.711179 8181 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:28.711210 8181 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.719228 8181 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42133
I20240503 07:59:28.719251 8308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42133 every 8 connection(s)
I20240503 07:59:28.720227 8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:28.727849 8309 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:28.728036 8309 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:28.728422 8309 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:28.729233 6740 ts_manager.cc:194] Re-registered known tserver with Master: 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133)
I20240503 07:59:28.729749 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:48719
I20240503 07:59:28.730330 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 8181
I20240503 07:59:28.730662 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:34081
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:43653
--webserver_port=35255
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:28.837448 8312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:28.837639 8312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:28.837679 8312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:28.840660 8312 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:28.840718 8312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:28.840792 8312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:28.844408 8312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:34081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=35255
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.8312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:28.845181 8312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:28.846218 8312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:28.854085 8317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.854642 8320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:28.855217 8318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:28.855800 8312 server_base.cc:1034] running on GCE node
I20240503 07:59:28.856135 8312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:28.856644 8312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:28.857784 8312 hybrid_clock.cc:648] HybridClock initialized: now 1714723168857770 us; error 26 us; skew 500 ppm
I20240503 07:59:28.859524 8312 webserver.cc:460] Webserver started at http://127.0.99.134:35255/ using document root <none> and password file <none>
I20240503 07:59:28.860036 8312 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:28.860108 8312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:28.863513 8312 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:28.865237 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.866286 8312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:28.866386 8312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ac6ca2734007406792d116f67b4fb000"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:28.866763 8312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:28.881839 8312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:28.882386 8312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:28.882532 8312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:28.883037 8312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:28.883842 8312 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:28.883888 8312 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.883927 8312 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:28.883955 8312 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:28.891472 8312 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34081
I20240503 07:59:28.891494 8439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34081 every 8 connection(s)
I20240503 07:59:28.892414 8312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:28.895385 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 8312
I20240503 07:59:28.895709 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:35189
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:43653
--webserver_port=40113
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:28.901134 8440 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:28.901342 8440 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:28.901757 8440 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:28.902707 6740 ts_manager.cc:194] Re-registered known tserver with Master: ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081)
I20240503 07:59:28.903470 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34185
W20240503 07:59:29.002255 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:29.002455 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:29.002501 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:29.005497 8443 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:29.005560 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:29.005625 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:29.009171 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:35189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=40113
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:29.009980 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:29.011046 8443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.019434 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.019968 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.020651 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.021222 8443 server_base.cc:1034] running on GCE node
I20240503 07:59:29.021603 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:29.022115 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:29.023258 8443 hybrid_clock.cc:648] HybridClock initialized: now 1714723169023242 us; error 30 us; skew 500 ppm
I20240503 07:59:29.025000 8443 webserver.cc:460] Webserver started at http://127.0.99.135:40113/ using document root <none> and password file <none>
I20240503 07:59:29.025539 8443 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.025612 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.028805 8443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:29.030535 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.031517 8443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.031618 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5629c2d36c84404dbef6e9e66860cc8d"
format_stamp: "Formatted at 2024-05-03 07:59:21 on dist-test-slave-11df"
I20240503 07:59:29.032063 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.041253 8443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.041785 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:29.041918 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.042387 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.043260 8443 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.043305 8443 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.043354 8443 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.043380 8443 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.051165 8443 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35189
I20240503 07:59:29.051180 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35189 every 8 connection(s)
I20240503 07:59:29.052127 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:29.060142 8571 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:29.060357 8571 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.060791 8571 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:29.060963 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 8443
I20240503 07:59:29.061101 398 test_util.cc:268] Using random seed: 425806061
I20240503 07:59:29.061753 6740 ts_manager.cc:194] Re-registered known tserver with Master: 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189)
I20240503 07:59:29.062476 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:39239
I20240503 07:59:29.070567 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 07:59:29.070777 398 test_util.cc:268] Using random seed: 425815735
I20240503 07:59:29.079145 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 07:59:29.079294 398 test_util.cc:268] Using random seed: 425824253
I20240503 07:59:29.089643 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 07:59:29.089794 398 test_util.cc:268] Using random seed: 425834752
I20240503 07:59:29.101709 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 07:59:29.101862 398 test_util.cc:268] Using random seed: 425846820
I20240503 07:59:29.120770 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 07:59:29.507596 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.523677 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:29.552060 8178 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:29.566234 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.570957 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:29.578037 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.580638 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:29.592821 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.731529 8309 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:29.903198 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:29.905522 8440 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:29.909816 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.913456 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.915215 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:29.936406 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:29.943063 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.959234 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:30.064424 8571 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:30.084277 7020 consensus_queue.cc:237] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:30.086819 7147 raft_consensus.cc:1261] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 07:59:30.087440 8598 consensus_queue.cc:1035] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 07:59:30.090108 8598 raft_consensus.cc:2932] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:30.090842 7150 raft_consensus.cc:2932] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:30.093962 6740 catalog_manager.cc:5484] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index -1 to 39, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
W20240503 07:59:30.108407 6956 consensus_peers.cc:485] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 -> Peer 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189): Couldn't send request to peer 5629c2d36c84404dbef6e9e66860cc8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f2f2ab1fe39408e8ee235acee496da7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.179894 6888 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
W20240503 07:59:30.188552 6822 consensus_peers.cc:485] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d -> Peer 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133): Couldn't send request to peer 6dc3aabcda15467498f2d0f01bbececf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829a4c6cc5d4441d907509c203635672. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.189277 7020 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 07:59:30.192778 7757 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:30.196518 8627 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.199024 6743 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 56, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.201973 7020 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.286563 7149 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:30.310873 6887 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240503 07:59:30.316520 8649 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
W20240503 07:59:30.318280 7087 consensus_peers.cc:485] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad -> Peer faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347): Couldn't send request to peer faca354acbbb49c89f3bbf9b3798eae6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce487b8bb9694381923b19d4e48257c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.321296 8640 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.323807 6740 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index -1 to 52, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.325122 6889 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.417454 6889 consensus_queue.cc:237] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:30.420974 7148 raft_consensus.cc:1261] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:30.422196 8682 consensus_queue.cc:1035] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 07:59:30.423902 8682 raft_consensus.cc:2932] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
W20240503 07:59:30.425621 6824 consensus_peers.cc:485] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d -> Peer ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): Couldn't send request to peer ac6ca2734007406792d116f67b4fb000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73029d928463492abf2ba405ce28763c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.426160 7149 raft_consensus.cc:2932] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:30.426980 6740 catalog_manager.cc:5484] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 50, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:30.508670 7020 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:30.511041 6887 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 07:59:30.511469 8598 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
W20240503 07:59:30.512382 6956 consensus_peers.cc:485] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 -> Peer 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189): Couldn't send request to peer 5629c2d36c84404dbef6e9e66860cc8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e21fe5b5f5a04168850a773226658450. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.521404 8598 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:30.522347 6887 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:30.523988 6743 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index -1 to 55, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:30.591385 6888 consensus_queue.cc:237] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:30.593305 7020 raft_consensus.cc:1261] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:30.596112 8627 consensus_queue.cc:1035] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
W20240503 07:59:30.597373 6825 consensus_peers.cc:485] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d -> Peer faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347): Couldn't send request to peer faca354acbbb49c89f3bbf9b3798eae6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 746df8ea7adb4b8cbd2b1782762683e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.598759 8627 raft_consensus.cc:2932] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.601125 6747 catalog_manager.cc:5484] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 50, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.606340 7019 raft_consensus.cc:2932] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:30.669193 8866 ts_tablet_manager.cc:902] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:30.672960 6889 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
W20240503 07:59:30.674749 6822 consensus_peers.cc:485] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d -> Peer 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133): Couldn't send request to peer 6dc3aabcda15467498f2d0f01bbececf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63ed07cc90364f058eb5ded00d3f7514. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.675496 7020 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 07:59:30.677037 8627 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 07:59:30.678632 8866 tablet_copy_client.cc:308] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:30.679947 8605 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.47: config changed from index -1 to 47, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.680912 7019 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.683369 6747 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 47, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:30.683974 8867 ts_tablet_manager.cc:902] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:30.684774 8867 tablet_copy_client.cc:308] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:30.695880 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet 7f2f2ab1fe39408e8ee235acee496da7 from peer 5629c2d36c84404dbef6e9e66860cc8d ({username='slave'} at 127.0.99.135:53919)
I20240503 07:59:30.696030 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet 7f2f2ab1fe39408e8ee235acee496da7 from peer 5629c2d36c84404dbef6e9e66860cc8d at {username='slave'} at 127.0.99.135:53919: session id = 5629c2d36c84404dbef6e9e66860cc8d-7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:30.697477 7039 tablet_copy_source_session.cc:205] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:30.705992 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2f2ab1fe39408e8ee235acee496da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.708290 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 829a4c6cc5d4441d907509c203635672 from peer 6dc3aabcda15467498f2d0f01bbececf ({username='slave'} at 127.0.99.133:47519)
I20240503 07:59:30.708428 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 829a4c6cc5d4441d907509c203635672 from peer 6dc3aabcda15467498f2d0f01bbececf at {username='slave'} at 127.0.99.133:47519: session id = 6dc3aabcda15467498f2d0f01bbececf-829a4c6cc5d4441d907509c203635672
I20240503 07:59:30.709728 6909 tablet_copy_source_session.cc:205] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:30.714287 8867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829a4c6cc5d4441d907509c203635672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.720774 8866 tablet_copy_client.cc:783] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:30.721005 8866 tablet_copy_client.cc:647] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:30.723260 8866 tablet_copy_client.cc:523] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:30.724915 8866 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap starting.
I20240503 07:59:30.728940 8867 tablet_copy_client.cc:783] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:30.729133 8867 tablet_copy_client.cc:647] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:30.739611 8867 tablet_copy_client.cc:523] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:30.745257 8867 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap starting.
I20240503 07:59:30.770517 8867 log.cc:824] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:30.775074 8866 log.cc:824] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:30.844655 7019 consensus_queue.cc:237] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:30.847501 7147 raft_consensus.cc:1261] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 07:59:30.848618 8667 consensus_queue.cc:1035] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 07:59:30.850639 8667 raft_consensus.cc:2932] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:30.853807 7149 raft_consensus.cc:2932] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:30.854194 6743 catalog_manager.cc:5484] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
W20240503 07:59:30.871070 6955 consensus_peers.cc:485] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 -> Peer ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): Couldn't send request to peer ac6ca2734007406792d116f67b4fb000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a786879136e44b8d99311bec38c2f85d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.909117 8907 ts_tablet_manager.cc:902] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:30.910023 8907 tablet_copy_client.cc:308] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:30.931847 8914 ts_tablet_manager.cc:902] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:30.937685 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet ce487b8bb9694381923b19d4e48257c8 from peer faca354acbbb49c89f3bbf9b3798eae6 ({username='slave'} at 127.0.99.132:45455)
I20240503 07:59:30.937850 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet ce487b8bb9694381923b19d4e48257c8 from peer faca354acbbb49c89f3bbf9b3798eae6 at {username='slave'} at 127.0.99.132:45455: session id = faca354acbbb49c89f3bbf9b3798eae6-ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:30.939034 8914 tablet_copy_client.cc:308] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:30.939332 7171 tablet_copy_source_session.cc:205] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:30.942981 8907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce487b8bb9694381923b19d4e48257c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.949565 8907 tablet_copy_client.cc:783] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:30.949748 8907 tablet_copy_client.cc:647] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:30.951987 8907 tablet_copy_client.cc:523] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:30.954244 8907 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap starting.
I20240503 07:59:30.975591 8866 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=694 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:30.975999 8866 tablet_bootstrap.cc:492] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap complete.
I20240503 07:59:30.976352 8866 ts_tablet_manager.cc:1367] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent bootstrapping tablet: real 0.252s user 0.071s sys 0.004s
I20240503 07:59:30.977689 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 73029d928463492abf2ba405ce28763c from peer ac6ca2734007406792d116f67b4fb000 ({username='slave'} at 127.0.99.134:56573)
I20240503 07:59:30.977790 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 73029d928463492abf2ba405ce28763c from peer ac6ca2734007406792d116f67b4fb000 at {username='slave'} at 127.0.99.134:56573: session id = ac6ca2734007406792d116f67b4fb000-73029d928463492abf2ba405ce28763c
I20240503 07:59:30.978655 6909 tablet_copy_source_session.cc:205] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:30.986519 8907 log.cc:824] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:30.978905 8866 raft_consensus.cc:348] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:30.991688 8866 raft_consensus.cc:729] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5629c2d36c84404dbef6e9e66860cc8d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.992210 8866 consensus_queue.cc:260] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:30.993119 8866 ts_tablet_manager.cc:1398] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent starting tablet: real 0.017s user 0.001s sys 0.004s
I20240503 07:59:30.993237 7019 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
W20240503 07:59:30.995939 6956 consensus_peers.cc:485] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 -> Peer 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189): Couldn't send request to peer 5629c2d36c84404dbef6e9e66860cc8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f94cbfb150547038338fbc0d892c8a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:30.998761 7147 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:30.998968 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73029d928463492abf2ba405ce28763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.999433 8598 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:31.000281 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-7f2f2ab1fe39408e8ee235acee496da7 received from {username='slave'} at 127.0.99.135:53919
I20240503 07:59:31.000362 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-7f2f2ab1fe39408e8ee235acee496da7 on tablet 7f2f2ab1fe39408e8ee235acee496da7 with peer 5629c2d36c84404dbef6e9e66860cc8d
I20240503 07:59:31.002092 8867 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.002436 8867 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap complete.
I20240503 07:59:31.002760 8867 ts_tablet_manager.cc:1367] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Time spent bootstrapping tablet: real 0.258s user 0.083s sys 0.002s
I20240503 07:59:31.004051 7776 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.005226 8867 raft_consensus.cc:348] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.008008 8867 raft_consensus.cc:729] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3aabcda15467498f2d0f01bbececf, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.011018 6747 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index -1 to 69, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.011612 8867 consensus_queue.cc:260] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.012568 8867 ts_tablet_manager.cc:1398] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 07:59:31.012719 7149 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.019490 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session 6dc3aabcda15467498f2d0f01bbececf-829a4c6cc5d4441d907509c203635672 received from {username='slave'} at 127.0.99.133:47519
I20240503 07:59:31.019598 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session 6dc3aabcda15467498f2d0f01bbececf-829a4c6cc5d4441d907509c203635672 on tablet 829a4c6cc5d4441d907509c203635672 with peer 6dc3aabcda15467498f2d0f01bbececf
I20240503 07:59:31.025564 8914 tablet_copy_client.cc:783] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.025758 8914 tablet_copy_client.cc:647] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.027503 8914 tablet_copy_client.cc:523] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.029227 8914 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Bootstrap starting.
I20240503 07:59:31.066695 8914 log.cc:824] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:31.068485 8938 ts_tablet_manager.cc:902] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:31.069015 8938 tablet_copy_client.cc:308] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:31.090646 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 746df8ea7adb4b8cbd2b1782762683e2 from peer faca354acbbb49c89f3bbf9b3798eae6 ({username='slave'} at 127.0.99.132:60197)
I20240503 07:59:31.090755 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 746df8ea7adb4b8cbd2b1782762683e2 from peer faca354acbbb49c89f3bbf9b3798eae6 at {username='slave'} at 127.0.99.132:60197: session id = faca354acbbb49c89f3bbf9b3798eae6-746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:31.091637 6909 tablet_copy_source_session.cc:205] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.093382 8938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746df8ea7adb4b8cbd2b1782762683e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.095431 8938 tablet_copy_client.cc:783] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.095568 8938 tablet_copy_client.cc:647] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.101747 8938 tablet_copy_client.cc:523] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.104377 8938 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap starting.
I20240503 07:59:31.108763 8866 ts_tablet_manager.cc:902] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:31.109216 8866 tablet_copy_client.cc:308] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:31.113046 6888 consensus_queue.cc:237] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.113981 8907 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=75 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:31.114410 8907 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap complete.
I20240503 07:59:31.114742 8907 ts_tablet_manager.cc:1367] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Time spent bootstrapping tablet: real 0.161s user 0.085s sys 0.004s
W20240503 07:59:31.114773 6825 consensus_peers.cc:485] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d -> Peer faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347): Couldn't send request to peer faca354acbbb49c89f3bbf9b3798eae6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8381a34bdbd84c9ca7dbe8d5aca087f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:31.117272 8907 raft_consensus.cc:348] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.118602 8907 raft_consensus.cc:729] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca354acbbb49c89f3bbf9b3798eae6, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.119161 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet e21fe5b5f5a04168850a773226658450 from peer 5629c2d36c84404dbef6e9e66860cc8d ({username='slave'} at 127.0.99.135:53919)
I20240503 07:59:31.119122 8907 consensus_queue.cc:260] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.119244 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet e21fe5b5f5a04168850a773226658450 from peer 5629c2d36c84404dbef6e9e66860cc8d at {username='slave'} at 127.0.99.135:53919: session id = 5629c2d36c84404dbef6e9e66860cc8d-e21fe5b5f5a04168850a773226658450
I20240503 07:59:31.120003 8907 ts_tablet_manager.cc:1398] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240503 07:59:31.120604 7039 tablet_copy_source_session.cc:205] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.121892 7019 raft_consensus.cc:1261] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.124413 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session faca354acbbb49c89f3bbf9b3798eae6-ce487b8bb9694381923b19d4e48257c8 received from {username='slave'} at 127.0.99.132:45455
I20240503 07:59:31.124516 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session faca354acbbb49c89f3bbf9b3798eae6-ce487b8bb9694381923b19d4e48257c8 on tablet ce487b8bb9694381923b19d4e48257c8 with peer faca354acbbb49c89f3bbf9b3798eae6
I20240503 07:59:31.126545 8792 consensus_queue.cc:1035] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:31.126629 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21fe5b5f5a04168850a773226658450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.128690 8866 tablet_copy_client.cc:783] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.128823 8866 tablet_copy_client.cc:647] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.134956 8866 tablet_copy_client.cc:523] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.135237 8682 raft_consensus.cc:2932] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:31.136186 8866 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap starting.
I20240503 07:59:31.138267 6740 catalog_manager.cc:5484] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 2, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:31.142627 7020 raft_consensus.cc:2932] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:31.161832 8914 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:31.162364 8914 tablet_bootstrap.cc:492] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Bootstrap complete.
I20240503 07:59:31.162700 8914 ts_tablet_manager.cc:1367] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Time spent bootstrapping tablet: real 0.134s user 0.076s sys 0.009s
I20240503 07:59:31.166146 8914 raft_consensus.cc:348] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:31.167861 8914 raft_consensus.cc:729] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.168390 8914 consensus_queue.cc:260] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:31.170830 8914 ts_tablet_manager.cc:1398] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240503 07:59:31.176354 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session ac6ca2734007406792d116f67b4fb000-73029d928463492abf2ba405ce28763c received from {username='slave'} at 127.0.99.134:56573
I20240503 07:59:31.176425 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session ac6ca2734007406792d116f67b4fb000-73029d928463492abf2ba405ce28763c on tablet 73029d928463492abf2ba405ce28763c with peer ac6ca2734007406792d116f67b4fb000
I20240503 07:59:31.227780 7147 consensus_queue.cc:237] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.232326 6888 raft_consensus.cc:1261] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240503 07:59:31.234380 8653 consensus_queue.cc:1035] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240503 07:59:31.235582 8938 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=741 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.235908 8938 tablet_bootstrap.cc:492] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap complete.
I20240503 07:59:31.236155 8938 ts_tablet_manager.cc:1367] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Time spent bootstrapping tablet: real 0.132s user 0.066s sys 0.004s
I20240503 07:59:31.236824 8938 raft_consensus.cc:348] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.236934 8938 raft_consensus.cc:729] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca354acbbb49c89f3bbf9b3798eae6, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.237147 8938 consensus_queue.cc:260] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.239091 8938 ts_tablet_manager.cc:1398] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:31.239784 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session faca354acbbb49c89f3bbf9b3798eae6-746df8ea7adb4b8cbd2b1782762683e2 received from {username='slave'} at 127.0.99.132:60197
I20240503 07:59:31.239859 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session faca354acbbb49c89f3bbf9b3798eae6-746df8ea7adb4b8cbd2b1782762683e2 on tablet 746df8ea7adb4b8cbd2b1782762683e2 with peer faca354acbbb49c89f3bbf9b3798eae6
I20240503 07:59:31.241921 8525 raft_consensus.cc:1203] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.81] Dedup: 1.63->[1.64-1.81]
I20240503 07:59:31.249199 8654 raft_consensus.cc:2932] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
W20240503 07:59:31.250551 7084 consensus_peers.cc:485] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad -> Peer 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133): Couldn't send request to peer 6dc3aabcda15467498f2d0f01bbececf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0088b17e63d431eaa49ef33a4af3b04. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:31.252300 6747 catalog_manager.cc:5484] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index -1 to 68, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:31.253162 6886 raft_consensus.cc:2932] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:31.256186 8263 raft_consensus.cc:1203] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.88] Dedup: 1.71->[1.72-1.88]
I20240503 07:59:31.266623 8866 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=749 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.266884 8866 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap complete.
I20240503 07:59:31.267027 8914 ts_tablet_manager.cc:902] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:31.267068 8866 ts_tablet_manager.cc:1367] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent bootstrapping tablet: real 0.131s user 0.071s sys 0.003s
I20240503 07:59:31.267459 8914 tablet_copy_client.cc:308] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:31.267589 8866 raft_consensus.cc:348] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:31.267712 8866 raft_consensus.cc:729] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5629c2d36c84404dbef6e9e66860cc8d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.267843 8866 consensus_queue.cc:260] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:31.268420 8866 ts_tablet_manager.cc:1398] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.268836 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-e21fe5b5f5a04168850a773226658450 received from {username='slave'} at 127.0.99.135:53919
I20240503 07:59:31.268915 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-e21fe5b5f5a04168850a773226658450 on tablet e21fe5b5f5a04168850a773226658450 with peer 5629c2d36c84404dbef6e9e66860cc8d
I20240503 07:59:31.271869 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet a786879136e44b8d99311bec38c2f85d from peer ac6ca2734007406792d116f67b4fb000 ({username='slave'} at 127.0.99.134:32919)
I20240503 07:59:31.271956 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet a786879136e44b8d99311bec38c2f85d from peer ac6ca2734007406792d116f67b4fb000 at {username='slave'} at 127.0.99.134:32919: session id = ac6ca2734007406792d116f67b4fb000-a786879136e44b8d99311bec38c2f85d
I20240503 07:59:31.272924 7039 tablet_copy_source_session.cc:205] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.274153 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a786879136e44b8d99311bec38c2f85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.275872 8867 ts_tablet_manager.cc:902] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:31.276309 8867 tablet_copy_client.cc:308] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:31.276783 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 63ed07cc90364f058eb5ded00d3f7514 from peer 6dc3aabcda15467498f2d0f01bbececf ({username='slave'} at 127.0.99.133:47519)
I20240503 07:59:31.276872 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 63ed07cc90364f058eb5ded00d3f7514 from peer 6dc3aabcda15467498f2d0f01bbececf at {username='slave'} at 127.0.99.133:47519: session id = 6dc3aabcda15467498f2d0f01bbececf-63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:31.277722 6909 tablet_copy_source_session.cc:205] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.279706 8914 tablet_copy_client.cc:783] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.279860 8914 tablet_copy_client.cc:647] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.281196 8867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ed07cc90364f058eb5ded00d3f7514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.282308 8914 tablet_copy_client.cc:523] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.283632 8867 tablet_copy_client.cc:783] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.283758 8867 tablet_copy_client.cc:647] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.285696 8914 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Bootstrap starting.
I20240503 07:59:31.302330 8867 tablet_copy_client.cc:523] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.309267 8867 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap starting.
I20240503 07:59:31.353704 8128 raft_consensus.cc:1203] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.82] Dedup: 1.76->[1.77-1.82]
I20240503 07:59:31.406409 8394 raft_consensus.cc:1203] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.84] Dedup: 1.72->[1.73-1.84]
I20240503 07:59:31.409498 8914 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=95 ignored=0} inserts{seen=726 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.409778 8914 tablet_bootstrap.cc:492] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Bootstrap complete.
I20240503 07:59:31.409978 8914 ts_tablet_manager.cc:1367] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Time spent bootstrapping tablet: real 0.124s user 0.075s sys 0.007s
I20240503 07:59:31.410537 8914 raft_consensus.cc:348] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:31.410660 8914 raft_consensus.cc:729] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.410790 8914 consensus_queue.cc:260] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:31.411134 8914 ts_tablet_manager.cc:1398] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 07:59:31.411584 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session ac6ca2734007406792d116f67b4fb000-a786879136e44b8d99311bec38c2f85d received from {username='slave'} at 127.0.99.134:32919
I20240503 07:59:31.411659 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session ac6ca2734007406792d116f67b4fb000-a786879136e44b8d99311bec38c2f85d on tablet a786879136e44b8d99311bec38c2f85d with peer ac6ca2734007406792d116f67b4fb000
I20240503 07:59:31.435261 7147 consensus_queue.cc:237] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:31.437427 6889 raft_consensus.cc:1261] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 07:59:31.438014 8654 consensus_queue.cc:1035] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 07:59:31.440063 8654 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:31.441573 6889 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:31.442874 6743 catalog_manager.cc:5484] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index -1 to 82, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
W20240503 07:59:31.468897 7086 consensus_peers.cc:485] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad -> Peer ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): Couldn't send request to peer ac6ca2734007406792d116f67b4fb000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba2f6cdcc0704518a2931544ab969df8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:31.477998 8866 ts_tablet_manager.cc:902] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:31.478439 8866 tablet_copy_client.cc:308] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:31.483981 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet 7f94cbfb150547038338fbc0d892c8a6 from peer 5629c2d36c84404dbef6e9e66860cc8d ({username='slave'} at 127.0.99.135:53919)
I20240503 07:59:31.484093 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet 7f94cbfb150547038338fbc0d892c8a6 from peer 5629c2d36c84404dbef6e9e66860cc8d at {username='slave'} at 127.0.99.135:53919: session id = 5629c2d36c84404dbef6e9e66860cc8d-7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:31.484963 7039 tablet_copy_source_session.cc:205] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.485939 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f94cbfb150547038338fbc0d892c8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.488454 8866 tablet_copy_client.cc:783] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.488605 8866 tablet_copy_client.cc:647] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.519452 8866 tablet_copy_client.cc:523] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.520856 8866 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap starting.
I20240503 07:59:31.529959 8867 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=742 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.530236 8867 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap complete.
I20240503 07:59:31.530436 8867 ts_tablet_manager.cc:1367] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Time spent bootstrapping tablet: real 0.221s user 0.059s sys 0.000s
I20240503 07:59:31.531095 8867 raft_consensus.cc:348] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.531209 8867 raft_consensus.cc:729] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3aabcda15467498f2d0f01bbececf, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.531339 8867 consensus_queue.cc:260] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.531597 8867 ts_tablet_manager.cc:1398] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:31.532043 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session 6dc3aabcda15467498f2d0f01bbececf-63ed07cc90364f058eb5ded00d3f7514 received from {username='slave'} at 127.0.99.133:47519
I20240503 07:59:31.532119 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session 6dc3aabcda15467498f2d0f01bbececf-63ed07cc90364f058eb5ded00d3f7514 on tablet 63ed07cc90364f058eb5ded00d3f7514 with peer 6dc3aabcda15467498f2d0f01bbececf
I20240503 07:59:31.556717 8128 raft_consensus.cc:1203] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.78] Dedup: 1.69->[1.70-1.78]
I20240503 07:59:31.629047 6886 consensus_queue.cc:237] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:31.630335 7147 raft_consensus.cc:1261] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 07:59:31.630843 8682 consensus_queue.cc:1035] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240503 07:59:31.633172 8627 raft_consensus.cc:2932] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index -1 to 87, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.633699 7149 raft_consensus.cc:2932] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.635833 6747 catalog_manager.cc:5484] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index -1 to 87, VOTER 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:31.657763 8866 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=711 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.658054 8866 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap complete.
I20240503 07:59:31.658239 8866 ts_tablet_manager.cc:1367] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent bootstrapping tablet: real 0.137s user 0.061s sys 0.014s
I20240503 07:59:31.658706 8866 raft_consensus.cc:348] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:31.658820 8866 raft_consensus.cc:729] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5629c2d36c84404dbef6e9e66860cc8d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.659348 8866 consensus_queue.cc:260] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:31.659646 8866 ts_tablet_manager.cc:1398] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:31.660720 8525 raft_consensus.cc:1203] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.89] Dedup: 1.77->[1.78-1.89]
I20240503 07:59:31.661449 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-7f94cbfb150547038338fbc0d892c8a6 received from {username='slave'} at 127.0.99.135:53919
I20240503 07:59:31.661532 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-7f94cbfb150547038338fbc0d892c8a6 on tablet 7f94cbfb150547038338fbc0d892c8a6 with peer 5629c2d36c84404dbef6e9e66860cc8d
W20240503 07:59:31.665051 6825 consensus_peers.cc:485] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d -> Peer 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189): Couldn't send request to peer 5629c2d36c84404dbef6e9e66860cc8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61da585c42664c72b8624641b1dac5dd. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:31.718685 8938 ts_tablet_manager.cc:902] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:31.719189 8938 tablet_copy_client.cc:308] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:31.721652 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 from peer faca354acbbb49c89f3bbf9b3798eae6 ({username='slave'} at 127.0.99.132:60197)
I20240503 07:59:31.721750 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 from peer faca354acbbb49c89f3bbf9b3798eae6 at {username='slave'} at 127.0.99.132:60197: session id = faca354acbbb49c89f3bbf9b3798eae6-8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:31.722528 6909 tablet_copy_source_session.cc:205] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.723366 8938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.725522 8938 tablet_copy_client.cc:783] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.725656 8938 tablet_copy_client.cc:647] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.730679 8867 ts_tablet_manager.cc:902] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:31.731148 8867 tablet_copy_client.cc:308] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:31.732357 8938 tablet_copy_client.cc:523] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.733649 8938 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap starting.
I20240503 07:59:31.746882 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet f0088b17e63d431eaa49ef33a4af3b04 from peer 6dc3aabcda15467498f2d0f01bbececf ({username='slave'} at 127.0.99.133:58907)
I20240503 07:59:31.747009 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet f0088b17e63d431eaa49ef33a4af3b04 from peer 6dc3aabcda15467498f2d0f01bbececf at {username='slave'} at 127.0.99.133:58907: session id = 6dc3aabcda15467498f2d0f01bbececf-f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:31.747992 7171 tablet_copy_source_session.cc:205] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:31.749111 8867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0088b17e63d431eaa49ef33a4af3b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.751194 8867 tablet_copy_client.cc:783] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:31.751334 8867 tablet_copy_client.cc:647] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:31.764117 8938 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.764377 8938 tablet_bootstrap.cc:492] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap complete.
I20240503 07:59:31.764591 8938 ts_tablet_manager.cc:1367] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 07:59:31.766826 8867 tablet_copy_client.cc:523] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:31.770071 8867 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap starting.
I20240503 07:59:31.765074 8938 raft_consensus.cc:348] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.780124 8938 raft_consensus.cc:729] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca354acbbb49c89f3bbf9b3798eae6, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.780297 8938 consensus_queue.cc:260] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:31.780591 8938 ts_tablet_manager.cc:1398] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:31.781078 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session faca354acbbb49c89f3bbf9b3798eae6-8381a34bdbd84c9ca7dbe8d5aca087f2 received from {username='slave'} at 127.0.99.132:60197
I20240503 07:59:31.781148 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session faca354acbbb49c89f3bbf9b3798eae6-8381a34bdbd84c9ca7dbe8d5aca087f2 on tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 with peer faca354acbbb49c89f3bbf9b3798eae6
I20240503 07:59:31.825686 8263 raft_consensus.cc:1203] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.79] Dedup: 1.57->[1.58-1.79]
I20240503 07:59:31.870469 8394 raft_consensus.cc:1203] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.81->[1.82-1.109] Dedup: 1.95->[1.96-1.109]
I20240503 07:59:31.895884 8867 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=757 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:31.896144 8867 tablet_bootstrap.cc:492] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap complete.
I20240503 07:59:31.896306 8867 ts_tablet_manager.cc:1367] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Time spent bootstrapping tablet: real 0.128s user 0.067s sys 0.013s
I20240503 07:59:31.896737 8867 raft_consensus.cc:348] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.896894 8867 raft_consensus.cc:729] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3aabcda15467498f2d0f01bbececf, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.897030 8867 consensus_queue.cc:260] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:31.899027 8867 ts_tablet_manager.cc:1398] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:31.910905 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session 6dc3aabcda15467498f2d0f01bbececf-f0088b17e63d431eaa49ef33a4af3b04 received from {username='slave'} at 127.0.99.133:58907
I20240503 07:59:31.911016 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session 6dc3aabcda15467498f2d0f01bbececf-f0088b17e63d431eaa49ef33a4af3b04 on tablet f0088b17e63d431eaa49ef33a4af3b04 with peer 6dc3aabcda15467498f2d0f01bbececf
I20240503 07:59:31.946429 9099 ts_tablet_manager.cc:902] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:31.947353 9099 tablet_copy_client.cc:308] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:31.997659 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.013235 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet ba2f6cdcc0704518a2931544ab969df8 from peer ac6ca2734007406792d116f67b4fb000 ({username='slave'} at 127.0.99.134:43609)
I20240503 07:59:32.013367 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet ba2f6cdcc0704518a2931544ab969df8 from peer ac6ca2734007406792d116f67b4fb000 at {username='slave'} at 127.0.99.134:43609: session id = ac6ca2734007406792d116f67b4fb000-ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:32.014395 7171 tablet_copy_source_session.cc:205] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.016033 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2f6cdcc0704518a2931544ab969df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.018245 9099 tablet_copy_client.cc:783] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.018393 9099 tablet_copy_client.cc:647] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.019706 8525 raft_consensus.cc:1203] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.92] Dedup: 1.80->[1.81-1.92]
I20240503 07:59:32.035311 8866 ts_tablet_manager.cc:902] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:32.035828 8866 tablet_copy_client.cc:308] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:32.038156 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.038841 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.042182 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.045995 9099 tablet_copy_client.cc:523] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.046085 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:32.047482 9099 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Bootstrap starting.
I20240503 07:59:32.053928 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 61da585c42664c72b8624641b1dac5dd from peer 5629c2d36c84404dbef6e9e66860cc8d ({username='slave'} at 127.0.99.135:48987)
I20240503 07:59:32.054034 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 61da585c42664c72b8624641b1dac5dd from peer 5629c2d36c84404dbef6e9e66860cc8d at {username='slave'} at 127.0.99.135:48987: session id = 5629c2d36c84404dbef6e9e66860cc8d-61da585c42664c72b8624641b1dac5dd
I20240503 07:59:32.054893 6909 tablet_copy_source_session.cc:205] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:32.055709 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61da585c42664c72b8624641b1dac5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.058180 8866 tablet_copy_client.cc:783] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:32.058331 8866 tablet_copy_client.cc:647] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:32.061110 8866 tablet_copy_client.cc:523] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:32.062441 8866 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap starting.
I20240503 07:59:32.070319 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:32.079703 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:32.093667 8132 raft_consensus.cc:1203] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:32.187717 8866 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=92 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:32.188234 8866 tablet_bootstrap.cc:492] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Bootstrap complete.
I20240503 07:59:32.188432 8866 ts_tablet_manager.cc:1367] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Time spent bootstrapping tablet: real 0.126s user 0.072s sys 0.012s
I20240503 07:59:32.188882 8866 raft_consensus.cc:348] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:32.189458 8866 raft_consensus.cc:729] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5629c2d36c84404dbef6e9e66860cc8d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.189612 8866 consensus_queue.cc:260] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:32.189884 8866 ts_tablet_manager.cc:1398] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.191107 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-61da585c42664c72b8624641b1dac5dd received from {username='slave'} at 127.0.99.135:48987
I20240503 07:59:32.191190 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session 5629c2d36c84404dbef6e9e66860cc8d-61da585c42664c72b8624641b1dac5dd on tablet 61da585c42664c72b8624641b1dac5dd with peer 5629c2d36c84404dbef6e9e66860cc8d
I20240503 07:59:32.214979 8262 raft_consensus.cc:1203] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.92] Dedup: 1.85->[1.86-1.92]
I20240503 07:59:32.273762 9099 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=0 applied=96 ignored=0} inserts{seen=781 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:32.274072 9099 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Bootstrap complete.
I20240503 07:59:32.274272 9099 ts_tablet_manager.cc:1367] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Time spent bootstrapping tablet: real 0.227s user 0.078s sys 0.010s
I20240503 07:59:32.274739 9099 raft_consensus.cc:348] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:32.274848 9099 raft_consensus.cc:729] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.275015 9099 consensus_queue.cc:260] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 1.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:32.276938 9099 ts_tablet_manager.cc:1398] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.279314 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session ac6ca2734007406792d116f67b4fb000-ba2f6cdcc0704518a2931544ab969df8 received from {username='slave'} at 127.0.99.134:43609
I20240503 07:59:32.279421 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session ac6ca2734007406792d116f67b4fb000-ba2f6cdcc0704518a2931544ab969df8 on tablet ba2f6cdcc0704518a2931544ab969df8 with peer ac6ca2734007406792d116f67b4fb000
I20240503 07:59:32.357591 8394 raft_consensus.cc:1203] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.81->[1.82-1.99] Dedup: 1.96->[1.97-1.99]
I20240503 07:59:32.469321 8523 raft_consensus.cc:1203] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.86->[1.87-1.102] Dedup: 1.93->[1.94-1.102]
I20240503 07:59:32.526508 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:32.533035 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.538317 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.559435 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:32.589367 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:32.627534 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.654796 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.834559 7146 consensus_queue.cc:237] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:32.841229 8263 raft_consensus.cc:1261] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 109. (index mismatch)
I20240503 07:59:32.841676 8653 consensus_queue.cc:1035] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 07:59:32.844242 8649 raft_consensus.cc:2932] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 68 to 108, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:32.848192 6747 catalog_manager.cc:5484] T f0088b17e63d431eaa49ef33a4af3b04 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 68 to 108, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:32.850554 8263 raft_consensus.cc:2932] T f0088b17e63d431eaa49ef33a4af3b04 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 68 to 108, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:32.912849 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet f0088b17e63d431eaa49ef33a4af3b04 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 108) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:32.963671 9221 tablet_replica.cc:330] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:32.967279 9221 raft_consensus.cc:2229] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.967578 9221 raft_consensus.cc:2258] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.020988 9221 ts_tablet_manager.cc:1875] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:33.026233 9221 ts_tablet_manager.cc:1888] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.106
I20240503 07:59:33.028851 9221 log.cc:1196] T f0088b17e63d431eaa49ef33a4af3b04 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:33.030611 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet f0088b17e63d431eaa49ef33a4af3b04 (table rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]) successfully deleted
I20240503 07:59:33.259526 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:33.264369 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:33.271986 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:33.288560 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:33.298703 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:33.335605 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:33.353665 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:33.444259 6886 consensus_queue.cc:237] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:33.445359 8128 raft_consensus.cc:1261] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:33.448132 9204 consensus_queue.cc:1035] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:33.482482 8627 raft_consensus.cc:2932] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:33.485371 6743 catalog_manager.cc:5484] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 2 to 3, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:33.483891 8132 raft_consensus.cc:2932] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:33.525233 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:33.532390 9286 tablet_replica.cc:330] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:33.532793 9286 raft_consensus.cc:2229] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.533028 9286 raft_consensus.cc:2258] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.541543 9286 ts_tablet_manager.cc:1875] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:33.568920 9286 ts_tablet_manager.cc:1888] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 07:59:33.569689 9286 log.cc:1196] T 8381a34bdbd84c9ca7dbe8d5aca087f2 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:33.570492 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet 8381a34bdbd84c9ca7dbe8d5aca087f2 (table TestTable [id=2d867d749b594723b00b88b6cd052156]) successfully deleted
I20240503 07:59:34.009903 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.017910 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.018258 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.067023 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.067734 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.080855 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.168655 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:34.379102 7018 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:34.380717 8524 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 07:59:34.381409 9033 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 07:59:34.384022 7776 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.150: config changed from index 69 to 150, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:34.384580 8523 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 69 to 150, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:34.386802 6747 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 69 to 150, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:34.422823 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f94cbfb150547038338fbc0d892c8a6 with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 150) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:34.438637 9355 tablet_replica.cc:330] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:34.440057 9355 raft_consensus.cc:2229] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.460981 9355 raft_consensus.cc:2258] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.467845 9355 ts_tablet_manager.cc:1875] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:34.471077 9355 ts_tablet_manager.cc:1888] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.149
I20240503 07:59:34.471453 9355 log.cc:1196] T 7f94cbfb150547038338fbc0d892c8a6 P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:34.472262 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet 7f94cbfb150547038338fbc0d892c8a6 (table rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]) successfully deleted
I20240503 07:59:34.731424 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.744396 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:34.746673 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.787084 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.797477 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.816107 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.847636 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.131206 7019 consensus_queue.cc:237] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.211, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:35.144105 8392 raft_consensus.cc:1261] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 07:59:35.144642 8662 consensus_queue.cc:1035] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 07:59:35.148912 9287 raft_consensus.cc:2932] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.212: config changed from index 82 to 212, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:35.151333 8394 raft_consensus.cc:2932] T a786879136e44b8d99311bec38c2f85d P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Committing config change with OpId 1.212: config changed from index 82 to 212, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:35.152917 6743 catalog_manager.cc:5484] T a786879136e44b8d99311bec38c2f85d P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 82 to 212, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:35.170976 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet a786879136e44b8d99311bec38c2f85d with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 212) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:35.181831 9433 tablet_replica.cc:330] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:35.182044 9433 raft_consensus.cc:2229] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.182250 9433 raft_consensus.cc:2258] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.183061 9433 ts_tablet_manager.cc:1875] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:35.206581 9433 ts_tablet_manager.cc:1888] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.211
I20240503 07:59:35.206766 9433 log.cc:1196] T a786879136e44b8d99311bec38c2f85d P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a786879136e44b8d99311bec38c2f85d
I20240503 07:59:35.207934 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet a786879136e44b8d99311bec38c2f85d (table rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]) successfully deleted
I20240503 07:59:35.368870 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.397635 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:35.412454 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:35.431126 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.432219 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.473268 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.506857 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:35.593531 6889 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514"
dest_uuid: "625353768527407d8849faefd6527f2d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53520
I20240503 07:59:35.593716 6889 raft_consensus.cc:595] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:35.630216 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.636540 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.640235 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.644614 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.651454 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.660632 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.667944 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.678038 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.692034 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.706709 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.726976 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.741412 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.756070 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.773815 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.795733 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.813607 6847 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.838415 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
I20240503 07:59:35.846578 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.851797 8239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.853638 6868 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:35.855422 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:35.856721 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:35.857095 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:35.859769 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:35.866386 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.890417 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.912535 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.938777 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
W20240503 07:59:35.962425 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Service unavailable: leader transfer in progress
I20240503 07:59:35.983059 8605 raft_consensus.cc:979] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: : Instructing follower 6dc3aabcda15467498f2d0f01bbececf to start an election
I20240503 07:59:35.983176 8605 raft_consensus.cc:1067] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Signalling peer 6dc3aabcda15467498f2d0f01bbececf to start an election
I20240503 07:59:35.983777 8262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514"
dest_uuid: "6dc3aabcda15467498f2d0f01bbececf"
from {username='slave'} at 127.0.99.129:53835
I20240503 07:59:35.983949 8262 raft_consensus.cc:482] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:35.984031 8262 raft_consensus.cc:3037] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:35.985054 8262 raft_consensus.cc:504] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:35.987087 8262 leader_election.cc:288] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [CANDIDATE]: Term 2 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735), 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:35.987365 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514" candidate_uuid: "6dc3aabcda15467498f2d0f01bbececf" candidate_term: 2 candidate_status { last_received { term: 1 index: 189 } } ignore_live_leader: true dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:35.987490 6889 raft_consensus.cc:3032] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:35.987530 6889 raft_consensus.cc:729] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:35.987692 6889 consensus_queue.cc:260] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:35.987865 6889 raft_consensus.cc:3037] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:35.988868 6889 raft_consensus.cc:2454] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc3aabcda15467498f2d0f01bbececf in term 2.
I20240503 07:59:35.989581 8199 leader_election.cc:302] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625353768527407d8849faefd6527f2d, 6dc3aabcda15467498f2d0f01bbececf; no voters:
W20240503 07:59:35.989769 6848 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54950: Illegal state: replica 625353768527407d8849faefd6527f2d is not leader of this config: current role FOLLOWER
I20240503 07:59:35.989996 9516 raft_consensus.cc:2781] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:35.990173 9516 raft_consensus.cc:686] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 LEADER]: Becoming Leader. State: Replica: 6dc3aabcda15467498f2d0f01bbececf, State: Running, Role: LEADER
I20240503 07:59:35.990765 9516 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:35.995306 6743 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf reported cstate change: term changed from 1 to 2, leader changed from 625353768527407d8849faefd6527f2d (127.0.99.129) to 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133). New cstate: current_term: 2 leader_uuid: "6dc3aabcda15467498f2d0f01bbececf" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:35.996577 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ed07cc90364f058eb5ded00d3f7514" candidate_uuid: "6dc3aabcda15467498f2d0f01bbececf" candidate_term: 2 candidate_status { last_received { term: 1 index: 189 } } ignore_live_leader: true dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:35.996691 7020 raft_consensus.cc:3037] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:35.997728 7020 raft_consensus.cc:2454] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc3aabcda15467498f2d0f01bbececf in term 2.
W20240503 07:59:36.018765 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44622: Illegal state: replica 76091450575f4076bd09550833d46f68 is not leader of this config: current role FOLLOWER
I20240503 07:59:36.075819 7020 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Refusing update from remote peer 6dc3aabcda15467498f2d0f01bbececf: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 2 index: 191. (index mismatch)
I20240503 07:59:36.077407 6889 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Refusing update from remote peer 6dc3aabcda15467498f2d0f01bbececf: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 2 index: 191. (index mismatch)
I20240503 07:59:36.078226 9517 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240503 07:59:36.080372 9516 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240503 07:59:36.083465 9146 mvcc.cc:205] Tried to move back new op lower bound from 7023506129196118016 to 7023506128862744576. Current Snapshot: MvccSnapshot[applied={T|T < 7023506129196118016 or (T in {7023506129196118016})}]
I20240503 07:59:36.301509 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:36.331722 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:36.332857 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:36.334254 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:36.356074 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:36.379901 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:36.409057 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:36.667197 6889 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.211, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:36.668613 8262 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 212. (index mismatch)
I20240503 07:59:36.669673 8605 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 07:59:36.675568 8605 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.212: config changed from index 56 to 212, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:36.676266 8262 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Committing config change with OpId 1.212: config changed from index 56 to 212, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:36.679006 6743 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 56 to 212, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:36.746403 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 829a4c6cc5d4441d907509c203635672 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 212) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:36.748040 9594 tablet_replica.cc:330] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:36.754734 9594 raft_consensus.cc:2229] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.754878 9594 raft_consensus.cc:2258] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.755641 9594 ts_tablet_manager.cc:1875] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:36.766419 9594 ts_tablet_manager.cc:1888] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.211
I20240503 07:59:36.766537 9594 log.cc:1196] T 829a4c6cc5d4441d907509c203635672 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/829a4c6cc5d4441d907509c203635672
I20240503 07:59:36.767112 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet 829a4c6cc5d4441d907509c203635672 (table rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]) successfully deleted
I20240503 07:59:36.947364 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:36.952932 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:36.962414 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:36.963150 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:36.968165 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.051718 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:37.108063 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:37.297515 6885 consensus_queue.cc:237] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:37.299825 8524 raft_consensus.cc:1261] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:37.300360 8792 consensus_queue.cc:1035] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:37.309384 8792 raft_consensus.cc:2932] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 87 to 246, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:37.311942 6743 catalog_manager.cc:5484] T 61da585c42664c72b8624641b1dac5dd P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 87 to 246, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:37.314047 8524 raft_consensus.cc:2932] T 61da585c42664c72b8624641b1dac5dd P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 87 to 246, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:37.339112 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 61da585c42664c72b8624641b1dac5dd with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 246) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:37.339411 9660 tablet_replica.cc:330] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:37.339599 9660 raft_consensus.cc:2229] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.339769 9660 raft_consensus.cc:2258] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.342522 9660 ts_tablet_manager.cc:1875] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:37.344815 9660 ts_tablet_manager.cc:1888] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.245
I20240503 07:59:37.344938 9660 log.cc:1196] T 61da585c42664c72b8624641b1dac5dd P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/61da585c42664c72b8624641b1dac5dd
I20240503 07:59:37.345510 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet 61da585c42664c72b8624641b1dac5dd (table rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]) successfully deleted
I20240503 07:59:37.463577 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:37.464478 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.472680 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:37.493131 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:37.495025 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.503522 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:37.562506 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.667886 6886 consensus_queue.cc:237] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.277, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:37.669391 8391 raft_consensus.cc:1261] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 278. (index mismatch)
I20240503 07:59:37.670639 7757 consensus_queue.cc:1035] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20240503 07:59:37.673627 7757 raft_consensus.cc:2932] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.278: config changed from index 50 to 278, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:37.676481 8394 raft_consensus.cc:2932] T 73029d928463492abf2ba405ce28763c P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Committing config change with OpId 1.278: config changed from index 50 to 278, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:37.676854 6738 catalog_manager.cc:5484] T 73029d928463492abf2ba405ce28763c P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 50 to 278, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:37.682214 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 73029d928463492abf2ba405ce28763c with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 278) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:37.682417 9660 tablet_replica.cc:330] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:37.682547 9660 raft_consensus.cc:2229] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.682657 9660 raft_consensus.cc:2258] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.683580 9660 ts_tablet_manager.cc:1875] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:37.685765 9660 ts_tablet_manager.cc:1888] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.277
I20240503 07:59:37.685832 9660 log.cc:1196] T 73029d928463492abf2ba405ce28763c P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/73029d928463492abf2ba405ce28763c
I20240503 07:59:37.686888 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet 73029d928463492abf2ba405ce28763c (table rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]) successfully deleted
I20240503 07:59:37.994158 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:38.014607 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:38.026166 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:38.053653 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:38.062534 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:38.081715 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:38.142447 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:38.404001 7018 consensus_queue.cc:237] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:38.407728 8523 raft_consensus.cc:1261] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 07:59:38.409116 7776 consensus_queue.cc:1035] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 292, Time since last communication: 0.000s
I20240503 07:59:38.416121 7776 raft_consensus.cc:2932] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 39 to 294, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:38.419411 6747 catalog_manager.cc:5484] T 7f2f2ab1fe39408e8ee235acee496da7 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 39 to 294, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:38.422622 8525 raft_consensus.cc:2932] T 7f2f2ab1fe39408e8ee235acee496da7 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 39 to 294, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:38.474982 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2f2ab1fe39408e8ee235acee496da7 with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 294) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:38.475324 9786 tablet_replica.cc:330] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:38.475538 9786 raft_consensus.cc:2229] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.503896 9786 pending_rounds.cc:62] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Trying to abort 1 pending ops.
I20240503 07:59:38.504091 9786 pending_rounds.cc:68] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Aborting op as it isn't in flight: id { term: 1 index: 293 } timestamp: 7023506138739154944 op_type: WRITE_OP write_request { tablet_id: "7f2f2ab1fe39408e8ee235acee496da7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\022\216\022\263\316\241]\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ea0d89fe3dbd3087" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506138709745664 authz_token { token_data: "\010\215\265\322\261\006\"3\n\005slave\022*\n e19cd3a96f5749c48f2a9cf4c183bee9\020\001\030\001 \001(\001" signature: "<redacted>""(\001V\205\037\317:\241T\"\205\177\036\320\025\330\272\232\227\200\326\025\257\006\357\2759\370`\231\271#\235\003\250\035\374[\334\022\337\026\270_\005\311\002n:\005\014*\266\272|m\317\272\321\313$0\331p" signing_key_seq_num: 0 } } request_id { client_id: "2f0db92dcdb746bb99e0c14ffcae1f99" seq_no: 850 first_incomplete_seq_no: 850 attempt_no: 0 }
I20240503 07:59:38.504485 9786 raft_consensus.cc:2258] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.506246 9786 ts_tablet_manager.cc:1875] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:38.509012 9786 ts_tablet_manager.cc:1888] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.293
I20240503 07:59:38.509121 9786 log.cc:1196] T 7f2f2ab1fe39408e8ee235acee496da7 P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:38.510078 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet 7f2f2ab1fe39408e8ee235acee496da7 (table rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]) successfully deleted
I20240503 07:59:38.777798 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:38.786378 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:38.788327 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:38.806737 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:38.816529 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:38.816541 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:38.844339 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.052920 7018 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } }
I20240503 07:59:39.057364 8525 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 1 index: 260. (index mismatch)
I20240503 07:59:39.059711 7776 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20240503 07:59:39.063026 7776 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.260: config changed from index 55 to 260, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:39.063534 8524 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.260: config changed from index 55 to 260, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:39.065737 6738 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 55 to 260, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } }
I20240503 07:59:39.086006 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet e21fe5b5f5a04168850a773226658450 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 260) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:39.091136 9855 tablet_replica.cc:330] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:39.091360 9855 raft_consensus.cc:2229] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.093195 9855 raft_consensus.cc:2258] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.094009 9855 ts_tablet_manager.cc:1875] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.100404 9855 ts_tablet_manager.cc:1888] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.259
I20240503 07:59:39.100760 9855 log.cc:1196] T e21fe5b5f5a04168850a773226658450 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e21fe5b5f5a04168850a773226658450
I20240503 07:59:39.101441 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet e21fe5b5f5a04168850a773226658450 (table rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]) successfully deleted
I20240503 07:59:39.330245 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:39.330366 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.334877 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.335747 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.337651 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:39.348022 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:39.359647 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:39.512667 6889 consensus_queue.cc:237] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:39.514432 8132 raft_consensus.cc:1261] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 07:59:39.515213 7757 consensus_queue.cc:1035] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240503 07:59:39.520716 8792 raft_consensus.cc:2932] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 50 to 296, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:39.521683 8132 raft_consensus.cc:2932] T 746df8ea7adb4b8cbd2b1782762683e2 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 50 to 296, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:39.523337 6743 catalog_manager.cc:5484] T 746df8ea7adb4b8cbd2b1782762683e2 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 50 to 296, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:39.527205 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 746df8ea7adb4b8cbd2b1782762683e2 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 296) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:39.528846 9914 tablet_replica.cc:330] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:39.529026 9914 raft_consensus.cc:2229] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.529512 9914 raft_consensus.cc:2258] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.530107 9914 ts_tablet_manager.cc:1875] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.544607 9914 ts_tablet_manager.cc:1888] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.295
I20240503 07:59:39.544682 9914 log.cc:1196] T 746df8ea7adb4b8cbd2b1782762683e2 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:39.545159 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet 746df8ea7adb4b8cbd2b1782762683e2 (table rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]) successfully deleted
I20240503 07:59:39.878340 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.892540 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:39.895727 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:39.897267 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:39.904374 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.916844 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:39.931936 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:40.110683 7150 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
W20240503 07:59:40.112017 8654 consensus_queue.cc:1159] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "625353768527407d8849faefd6527f2d" responder_term: 1 status { last_received { term: 1 index: 366 } last_committed_idx: 365 last_received_current_leader { term: 1 index: 366 } }
I20240503 07:59:40.116843 8128 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20240503 07:59:40.122789 8654 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20240503 07:59:40.125901 8654 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index 52 to 367, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:40.128683 6740 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 52 to 367, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:40.131268 8128 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index 52 to 367, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:40.147248 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet ce487b8bb9694381923b19d4e48257c8 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 367) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:40.147574 9985 tablet_replica.cc:330] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:40.147753 9985 raft_consensus.cc:2229] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.148134 9985 pending_rounds.cc:62] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Trying to abort 1 pending ops.
I20240503 07:59:40.148257 9985 pending_rounds.cc:68] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Aborting op as it isn't in flight: id { term: 1 index: 366 } timestamp: 7023506145684779008 op_type: WRITE_OP write_request { tablet_id: "ce487b8bb9694381923b19d4e48257c8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\233\305D]\213\327Y\316\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""82f4e28ab689e8bb" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506145632432128 authz_token { token_data: "\010\215\265\322\261\006\"3\n\005slave\022*\n e19cd3a96f5749c48f2a9cf4c183bee9\020\001\030\001 \001(\001" signature: "<redacted>""(\001V\205\037\317:\241T\"\205\177\036\320\025\330\272\232\227\200\326\025\257\006\357\2759\370`\231\271#\235\003\250\035\374[\334\022\337\026\270_\005\311\002n:\005\014*\266\272|m\317\272\321\313$0\331p" signing_key_seq_num: 0 } } request_id { client_id: "2f0db92dcdb746bb99e0c14ffcae1f99" seq_no: 1044 first_incomplete_seq_no: 1044 attempt_no: 0 }
I20240503 07:59:40.148648 9985 raft_consensus.cc:2258] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.149382 9985 ts_tablet_manager.cc:1875] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.152096 9985 ts_tablet_manager.cc:1888] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.366
I20240503 07:59:40.152462 9985 log.cc:1196] T ce487b8bb9694381923b19d4e48257c8 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:40.153313 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet ce487b8bb9694381923b19d4e48257c8 (table rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]) successfully deleted
I20240503 07:59:40.604574 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:40.607156 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:40.614267 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:40.632478 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:40.632827 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:40.671345 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:40.703271 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:40.877741 7148 consensus_queue.cc:237] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 1.371, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:40.878937 8391 raft_consensus.cc:1261] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 371. Preceding OpId from leader: term: 1 index: 372. (index mismatch)
I20240503 07:59:40.879767 8654 consensus_queue.cc:1035] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20240503 07:59:40.883503 9979 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.372: config changed from index 82 to 372, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:40.885571 8394 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Committing config change with OpId 1.372: config changed from index 82 to 372, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:40.886163 6743 catalog_manager.cc:5484] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 82 to 372, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:40.911592 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2f6cdcc0704518a2931544ab969df8 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 372) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:40.917665 10066 tablet_replica.cc:330] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:40.917883 10066 raft_consensus.cc:2229] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.918052 10066 raft_consensus.cc:2258] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.919107 10066 ts_tablet_manager.cc:1875] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.930518 10066 ts_tablet_manager.cc:1888] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.371
I20240503 07:59:40.930691 10066 log.cc:1196] T ba2f6cdcc0704518a2931544ab969df8 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:40.932736 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet ba2f6cdcc0704518a2931544ab969df8 (table rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]) successfully deleted
I20240503 07:59:41.365789 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.381155 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:41.383008 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.391888 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.404994 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.411324 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.474476 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.589363 8262 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 2.386, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:41.590909 7018 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Refusing update from remote peer 6dc3aabcda15467498f2d0f01bbececf: Log matching property violated. Preceding OpId in replica: term: 2 index: 386. Preceding OpId from leader: term: 2 index: 387. (index mismatch)
I20240503 07:59:41.593928 10017 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 07:59:41.599041 10017 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 LEADER]: Committing config change with OpId 2.387: config changed from index 47 to 387, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:41.601583 6743 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf reported cstate change: config changed from index 47 to 387, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6dc3aabcda15467498f2d0f01bbececf" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:41.604555 7018 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Committing config change with OpId 2.387: config changed from index 47 to 387, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:41.616533 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ed07cc90364f058eb5ded00d3f7514 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 387) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:41.635864 10157 tablet_replica.cc:330] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:41.636076 10157 raft_consensus.cc:2229] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:41.638016 10157 raft_consensus.cc:2258] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:41.641005 10157 ts_tablet_manager.cc:1875] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:41.647804 10157 ts_tablet_manager.cc:1888] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.386
I20240503 07:59:41.647950 10157 log.cc:1196] T 63ed07cc90364f058eb5ded00d3f7514 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:41.652480 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet 63ed07cc90364f058eb5ded00d3f7514 (table rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]) successfully deleted
I20240503 07:59:41.895339 7148 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 1.388, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:41.898190 6888 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
W20240503 07:59:41.899185 7087 consensus_peers.cc:485] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad -> Peer faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347): Couldn't send request to peer faca354acbbb49c89f3bbf9b3798eae6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a37812b3d0a346e3a1bc9c57956f2ca5. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:41.899458 10200 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240503 07:59:41.901727 8654 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.389: config changed from index -1 to 389, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:41.904194 6888 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index -1 to 389, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:41.904541 6747 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index -1 to 389, VOTER faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:42.106171 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:42.117537 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:42.118580 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.118710 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:42.127372 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.134980 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:42.169553 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.507589 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.517042 10302 ts_tablet_manager.cc:902] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:42.517611 10302 tablet_copy_client.cc:308] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:42.519376 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet a37812b3d0a346e3a1bc9c57956f2ca5 from peer faca354acbbb49c89f3bbf9b3798eae6 ({username='slave'} at 127.0.99.132:45455)
I20240503 07:59:42.519495 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet a37812b3d0a346e3a1bc9c57956f2ca5 from peer faca354acbbb49c89f3bbf9b3798eae6 at {username='slave'} at 127.0.99.132:45455: session id = faca354acbbb49c89f3bbf9b3798eae6-a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:42.520588 7171 tablet_copy_source_session.cc:205] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:42.521448 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37812b3d0a346e3a1bc9c57956f2ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.523871 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:42.525216 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.525250 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:42.527920 10302 tablet_copy_client.cc:783] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:42.528334 10302 tablet_copy_client.cc:647] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:42.529551 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:42.530536 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:42.535887 10302 tablet_copy_client.cc:523] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:42.537813 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:42.545022 10302 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap starting.
I20240503 07:59:42.975760 10302 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap replayed 1/1 log segments. Stats: ops{read=412 overwritten=0 applied=412 ignored=0} inserts{seen=1097 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:42.976190 10302 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Bootstrap complete.
I20240503 07:59:42.976531 10302 ts_tablet_manager.cc:1367] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Time spent bootstrapping tablet: real 0.432s user 0.279s sys 0.023s
I20240503 07:59:42.977082 10302 raft_consensus.cc:348] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:42.977286 10302 raft_consensus.cc:729] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca354acbbb49c89f3bbf9b3798eae6, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.977530 10302 consensus_queue.cc:260] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 412, Last appended: 1.412, Last appended by leader: 412, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:42.977978 10302 ts_tablet_manager.cc:1398] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:42.979235 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session faca354acbbb49c89f3bbf9b3798eae6-a37812b3d0a346e3a1bc9c57956f2ca5 received from {username='slave'} at 127.0.99.132:45455
I20240503 07:59:42.979326 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session faca354acbbb49c89f3bbf9b3798eae6-a37812b3d0a346e3a1bc9c57956f2ca5 on tablet a37812b3d0a346e3a1bc9c57956f2ca5 with peer faca354acbbb49c89f3bbf9b3798eae6
I20240503 07:59:42.983124 8131 raft_consensus.cc:1203] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.388->[1.389-1.422] Dedup: 1.412->[1.413-1.422]
I20240503 07:59:43.145412 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:43.148399 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:43.150081 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:43.156092 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:43.164209 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:43.166607 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:43.179955 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:43.287027 7150 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 437, Committed index: 437, Last appended: 1.437, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } }
I20240503 07:59:43.290721 8128 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 437. Preceding OpId from leader: term: 1 index: 438. (index mismatch)
I20240503 07:59:43.291743 8654 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 438, Last known committed idx: 437, Time since last communication: 0.000s
I20240503 07:59:43.293732 9174 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.438: config changed from index 389 to 438, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:43.295651 8128 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.438: config changed from index 389 to 438, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:43.296391 6747 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 389 to 438, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } }
I20240503 07:59:43.304525 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet a37812b3d0a346e3a1bc9c57956f2ca5 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 438) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:43.304847 10381 tablet_replica.cc:330] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:43.305114 10381 raft_consensus.cc:2229] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:43.305294 10381 raft_consensus.cc:2258] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:43.308297 10381 ts_tablet_manager.cc:1875] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:43.322769 10381 ts_tablet_manager.cc:1888] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.437
I20240503 07:59:43.322853 10381 log.cc:1196] T a37812b3d0a346e3a1bc9c57956f2ca5 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:43.323616 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet a37812b3d0a346e3a1bc9c57956f2ca5 (table rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]) successfully deleted
I20240503 07:59:43.843964 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:43.845373 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:43.849628 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:43.858068 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:43.861339 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:43.868144 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:43.879633 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.031801 8262 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 470, Committed index: 470, Last appended: 2.470, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:44.033222 7018 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Refusing update from remote peer 6dc3aabcda15467498f2d0f01bbececf: Log matching property violated. Preceding OpId in replica: term: 2 index: 470. Preceding OpId from leader: term: 2 index: 471. (index mismatch)
I20240503 07:59:44.033712 9905 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 471, Last known committed idx: 470, Time since last communication: 0.000s
I20240503 07:59:44.036895 9905 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 LEADER]: Committing config change with OpId 2.471: config changed from index 387 to 471, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:44.039453 6743 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf reported cstate change: config changed from index 387 to 471, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "6dc3aabcda15467498f2d0f01bbececf" committed_config { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:44.039714 7018 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Committing config change with OpId 2.471: config changed from index 387 to 471, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
W20240503 07:59:44.042642 8198 consensus_peers.cc:485] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf -> Peer ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): Couldn't send request to peer ac6ca2734007406792d116f67b4fb000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63ed07cc90364f058eb5ded00d3f7514. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:44.118846 7150 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 506, Committed index: 506, Last appended: 1.506, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
W20240503 07:59:44.120613 7084 consensus_peers.cc:485] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad -> Peer 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133): Couldn't send request to peer 6dc3aabcda15467498f2d0f01bbececf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce487b8bb9694381923b19d4e48257c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:44.120962 8131 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 506. Preceding OpId from leader: term: 1 index: 507. (index mismatch)
I20240503 07:59:44.122260 9795 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 507, Last known committed idx: 506, Time since last communication: 0.000s
I20240503 07:59:44.125279 10200 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.507: config changed from index 367 to 507, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:44.127938 6739 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 367 to 507, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:44.128329 8128 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.507: config changed from index 367 to 507, VOTER 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133) added. New config: { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:44.252796 7020 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 460, Committed index: 460, Last appended: 1.461, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:44.254117 8523 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 461. Preceding OpId from leader: term: 1 index: 462. (index mismatch)
W20240503 07:59:44.254361 6955 consensus_peers.cc:485] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 -> Peer ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): Couldn't send request to peer ac6ca2734007406792d116f67b4fb000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e21fe5b5f5a04168850a773226658450. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:44.256307 10147 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 462, Last known committed idx: 460, Time since last communication: 0.000s
I20240503 07:59:44.258647 10147 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.462: config changed from index 260 to 462, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:44.262190 6739 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 260 to 462, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:44.260630 8525 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.462: config changed from index 260 to 462, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) added. New config: { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:44.472692 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.483436 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.484475 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:44.503247 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:44.529819 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:44.537567 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.548719 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:44.596071 10531 ts_tablet_manager.cc:902] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Initiating tablet copy from peer 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133)
I20240503 07:59:44.596786 10531 tablet_copy_client.cc:308] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:42133
I20240503 07:59:44.624846 8283 tablet_copy_service.cc:140] P 6dc3aabcda15467498f2d0f01bbececf: Received BeginTabletCopySession request for tablet 63ed07cc90364f058eb5ded00d3f7514 from peer ac6ca2734007406792d116f67b4fb000 ({username='slave'} at 127.0.99.134:54649)
I20240503 07:59:44.625001 8283 tablet_copy_service.cc:161] P 6dc3aabcda15467498f2d0f01bbececf: Beginning new tablet copy session on tablet 63ed07cc90364f058eb5ded00d3f7514 from peer ac6ca2734007406792d116f67b4fb000 at {username='slave'} at 127.0.99.134:54649: session id = ac6ca2734007406792d116f67b4fb000-63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:44.626353 8283 tablet_copy_source_session.cc:205] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:44.627831 10531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ed07cc90364f058eb5ded00d3f7514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.630021 10531 tablet_copy_client.cc:783] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:44.630153 10531 tablet_copy_client.cc:647] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:44.635284 10531 tablet_copy_client.cc:523] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:44.636957 10531 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Bootstrap starting.
I20240503 07:59:44.733875 10557 ts_tablet_manager.cc:902] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:44.734459 10557 tablet_copy_client.cc:308] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:44.734898 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet ce487b8bb9694381923b19d4e48257c8 from peer 6dc3aabcda15467498f2d0f01bbececf ({username='slave'} at 127.0.99.133:58907)
I20240503 07:59:44.735014 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet ce487b8bb9694381923b19d4e48257c8 from peer 6dc3aabcda15467498f2d0f01bbececf at {username='slave'} at 127.0.99.133:58907: session id = 6dc3aabcda15467498f2d0f01bbececf-ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:44.735965 7171 tablet_copy_source_session.cc:205] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:44.740929 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce487b8bb9694381923b19d4e48257c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.746414 10557 tablet_copy_client.cc:783] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:44.746641 10557 tablet_copy_client.cc:647] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:44.753136 10557 tablet_copy_client.cc:523] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:44.754838 10557 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap starting.
I20240503 07:59:44.833765 10570 ts_tablet_manager.cc:902] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:44.837908 10570 tablet_copy_client.cc:308] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:44.838387 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet e21fe5b5f5a04168850a773226658450 from peer ac6ca2734007406792d116f67b4fb000 ({username='slave'} at 127.0.99.134:32919)
I20240503 07:59:44.838486 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet e21fe5b5f5a04168850a773226658450 from peer ac6ca2734007406792d116f67b4fb000 at {username='slave'} at 127.0.99.134:32919: session id = ac6ca2734007406792d116f67b4fb000-e21fe5b5f5a04168850a773226658450
I20240503 07:59:44.839323 7039 tablet_copy_source_session.cc:205] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:44.844703 10570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21fe5b5f5a04168850a773226658450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.847827 10570 tablet_copy_client.cc:783] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:44.849499 10570 tablet_copy_client.cc:647] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:44.863286 10570 tablet_copy_client.cc:523] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:44.864717 10570 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Bootstrap starting.
I20240503 07:59:45.034731 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.035058 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:45.047577 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:45.078866 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.099737 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:45.109318 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:45.110397 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.488304 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:45.496263 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.498204 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:45.523065 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.537758 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.566629 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:45.577219 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 07:59:45.600770 10570 log.cc:924] Time spent T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240503 07:59:45.684244 10531 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Bootstrap replayed 1/1 log segments. Stats: ops{read=501 overwritten=0 applied=501 ignored=0} inserts{seen=1183 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:45.690834 10531 tablet_bootstrap.cc:492] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Bootstrap complete.
I20240503 07:59:45.691167 10531 ts_tablet_manager.cc:1367] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Time spent bootstrapping tablet: real 1.054s user 0.339s sys 0.028s
I20240503 07:59:45.691690 10531 raft_consensus.cc:348] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:45.691886 10531 raft_consensus.cc:729] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.692066 10531 consensus_queue.cc:260] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 501, Last appended: 2.501, Last appended by leader: 501, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 471 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:45.694535 10531 ts_tablet_manager.cc:1398] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:45.697522 8283 tablet_copy_service.cc:342] P 6dc3aabcda15467498f2d0f01bbececf: Request end of tablet copy session ac6ca2734007406792d116f67b4fb000-63ed07cc90364f058eb5ded00d3f7514 received from {username='slave'} at 127.0.99.134:54649
I20240503 07:59:45.697633 8283 tablet_copy_service.cc:433] P 6dc3aabcda15467498f2d0f01bbececf: ending tablet copy session ac6ca2734007406792d116f67b4fb000-63ed07cc90364f058eb5ded00d3f7514 on tablet 63ed07cc90364f058eb5ded00d3f7514 with peer ac6ca2734007406792d116f67b4fb000
I20240503 07:59:45.838330 10570 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Bootstrap replayed 1/1 log segments. Stats: ops{read=471 overwritten=0 applied=471 ignored=0} inserts{seen=1141 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:45.838771 10570 tablet_bootstrap.cc:492] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Bootstrap complete.
I20240503 07:59:45.839062 10570 ts_tablet_manager.cc:1367] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Time spent bootstrapping tablet: real 0.974s user 0.322s sys 0.024s
I20240503 07:59:45.839545 10570 raft_consensus.cc:348] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:45.839742 10570 raft_consensus.cc:729] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.839937 10570 consensus_queue.cc:260] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 471, Last appended: 1.471, Last appended by leader: 471, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:45.841243 10570 ts_tablet_manager.cc:1398] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:45.842717 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session ac6ca2734007406792d116f67b4fb000-e21fe5b5f5a04168850a773226658450 received from {username='slave'} at 127.0.99.134:32919
I20240503 07:59:45.842797 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session ac6ca2734007406792d116f67b4fb000-e21fe5b5f5a04168850a773226658450 on tablet e21fe5b5f5a04168850a773226658450 with peer ac6ca2734007406792d116f67b4fb000
I20240503 07:59:45.872222 10557 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap replayed 1/1 log segments. Stats: ops{read=521 overwritten=0 applied=521 ignored=0} inserts{seen=1191 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:45.872601 10557 tablet_bootstrap.cc:492] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Bootstrap complete.
I20240503 07:59:45.872924 10557 ts_tablet_manager.cc:1367] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Time spent bootstrapping tablet: real 1.118s user 0.360s sys 0.026s
I20240503 07:59:45.873385 10557 raft_consensus.cc:348] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:45.873528 10557 raft_consensus.cc:729] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3aabcda15467498f2d0f01bbececf, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.873692 10557 consensus_queue.cc:260] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 521, Last appended: 1.521, Last appended by leader: 521, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:45.874001 10557 ts_tablet_manager.cc:1398] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.880981 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session 6dc3aabcda15467498f2d0f01bbececf-ce487b8bb9694381923b19d4e48257c8 received from {username='slave'} at 127.0.99.133:58907
I20240503 07:59:45.881078 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session 6dc3aabcda15467498f2d0f01bbececf-ce487b8bb9694381923b19d4e48257c8 on tablet ce487b8bb9694381923b19d4e48257c8 with peer 6dc3aabcda15467498f2d0f01bbececf
I20240503 07:59:45.886211 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:45.903290 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:45.906332 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:45.908015 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.917637 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.918620 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:45.930971 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.946069 8263 raft_consensus.cc:1203] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.506->[1.507-1.544] Dedup: 1.521->[1.522-1.544]
I20240503 07:59:45.977339 8394 raft_consensus.cc:1203] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.470->[2.471-2.543] Dedup: 2.501->[2.502-2.543]
I20240503 07:59:46.024681 8392 raft_consensus.cc:1203] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.461->[1.462-1.495] Dedup: 1.471->[1.472-1.495]
I20240503 07:59:46.275146 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:46.283586 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:46.285946 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:46.295482 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:46.307121 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:46.308570 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:46.333763 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:46.452889 8263 consensus_queue.cc:237] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 563, Committed index: 563, Last appended: 2.564, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:46.454607 8392 raft_consensus.cc:1261] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Refusing update from remote peer 6dc3aabcda15467498f2d0f01bbececf: Log matching property violated. Preceding OpId in replica: term: 2 index: 564. Preceding OpId from leader: term: 2 index: 565. (index mismatch)
I20240503 07:59:46.455313 10440 consensus_queue.cc:1035] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 565, Last known committed idx: 563, Time since last communication: 0.000s
I20240503 07:59:46.457087 10440 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf [term 2 LEADER]: Committing config change with OpId 2.565: config changed from index 471 to 565, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:46.462008 8392 raft_consensus.cc:2932] T 63ed07cc90364f058eb5ded00d3f7514 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Committing config change with OpId 2.565: config changed from index 471 to 565, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:46.462114 6743 catalog_manager.cc:5484] T 63ed07cc90364f058eb5ded00d3f7514 P 6dc3aabcda15467498f2d0f01bbececf reported cstate change: config changed from index 471 to 565, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6dc3aabcda15467498f2d0f01bbececf" committed_config { opid_index: 565 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:46.466064 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ed07cc90364f058eb5ded00d3f7514 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 565) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:46.466997 10750 tablet_replica.cc:330] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:46.467217 10750 raft_consensus.cc:2229] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:46.467383 10750 pending_rounds.cc:62] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Trying to abort 1 pending ops.
I20240503 07:59:46.467499 10750 pending_rounds.cc:68] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Aborting op as it isn't in flight: id { term: 2 index: 564 } timestamp: 7023506171704627200 op_type: WRITE_OP write_request { tablet_id: "63ed07cc90364f058eb5ded00d3f7514" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000T\331\030\000\325\344\334$\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ef597eb3b91aa5c3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506171675328512 authz_token { token_data: "\010\215\265\322\261\006\"3\n\005slave\022*\n c43e47b81f5a44e0807d5a175e5b28d7\020\001\030\001 \001(\001" signature: "<redacted>""O\t\250&\247\276I\305\263\305P\3556X\251\226\320\225\226\305\034-\020\013\225\372\354\350vG\032\232\252Z\207[\375\225\347z\333o\336\270\354\025<\\\031\367\202\376\341\310EC\322=\315[\255 7\360" signing_key_seq_num: 0 } } request_id { client_id: "7adf74a9a3824982867e0a8b263e158b" seq_no: 1612 first_incomplete_seq_no: 1612 attempt_no: 0 }
I20240503 07:59:46.468487 10750 raft_consensus.cc:2258] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:46.469292 10750 ts_tablet_manager.cc:1875] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:46.474579 10750 ts_tablet_manager.cc:1888] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.564
I20240503 07:59:46.475072 10750 log.cc:1196] T 63ed07cc90364f058eb5ded00d3f7514 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:46.475672 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet 63ed07cc90364f058eb5ded00d3f7514 (table rebalance_test_table_1 [id=c43e47b81f5a44e0807d5a175e5b28d7]) successfully deleted
I20240503 07:59:46.895972 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:46.906589 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:46.912984 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:46.933166 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:46.948869 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:46.983700 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:46.996105 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:47.081256 7018 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e21fe5b5f5a04168850a773226658450"
dest_uuid: "76091450575f4076bd09550833d46f68"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50240
I20240503 07:59:47.081439 7018 raft_consensus.cc:595] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:47.105942 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.108379 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.117902 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.125730 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.139108 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.146771 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.158605 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.169447 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.183086 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.194059 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.209264 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.225018 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.240764 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.259572 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
I20240503 07:59:47.274713 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:47.279261 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
I20240503 07:59:47.284158 6998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:47.287701 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:47.297907 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:47.300029 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:47.305009 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:47.306514 8374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:47.311460 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.333442 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.352412 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.373119 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.399442 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.427711 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.454344 6977 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
W20240503 07:59:47.480178 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Service unavailable: leader transfer in progress
I20240503 07:59:47.488543 10147 raft_consensus.cc:979] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: : Instructing follower ac6ca2734007406792d116f67b4fb000 to start an election
I20240503 07:59:47.488648 10147 raft_consensus.cc:1067] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Signalling peer ac6ca2734007406792d116f67b4fb000 to start an election
I20240503 07:59:47.491611 8394 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e21fe5b5f5a04168850a773226658450"
dest_uuid: "ac6ca2734007406792d116f67b4fb000"
from {username='slave'} at 127.0.99.130:58659
I20240503 07:59:47.491828 8394 raft_consensus.cc:482] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:47.491947 8394 raft_consensus.cc:3037] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:47.493077 8394 raft_consensus.cc:504] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:47.501019 8394 leader_election.cc:288] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [CANDIDATE]: Term 2 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735), 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189)
I20240503 07:59:47.503872 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21fe5b5f5a04168850a773226658450" candidate_uuid: "ac6ca2734007406792d116f67b4fb000" candidate_term: 2 candidate_status { last_received { term: 1 index: 537 } } ignore_live_leader: true dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:47.503989 7018 raft_consensus.cc:3032] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:47.504042 7018 raft_consensus.cc:729] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:47.504168 7018 consensus_queue.cc:260] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 537, Committed index: 537, Last appended: 1.537, Last appended by leader: 537, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:47.504335 7018 raft_consensus.cc:3037] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:47.505374 7018 raft_consensus.cc:2454] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6ca2734007406792d116f67b4fb000 in term 2.
I20240503 07:59:47.506227 8328 leader_election.cc:302] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76091450575f4076bd09550833d46f68, ac6ca2734007406792d116f67b4fb000; no voters:
W20240503 07:59:47.508692 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Illegal state: replica 76091450575f4076bd09550833d46f68 is not leader of this config: current role FOLLOWER
I20240503 07:59:47.511961 8523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21fe5b5f5a04168850a773226658450" candidate_uuid: "ac6ca2734007406792d116f67b4fb000" candidate_term: 2 candidate_status { last_received { term: 1 index: 537 } } ignore_live_leader: true dest_uuid: "5629c2d36c84404dbef6e9e66860cc8d"
I20240503 07:59:47.512140 8523 raft_consensus.cc:3037] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:47.513249 8523 raft_consensus.cc:2454] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6ca2734007406792d116f67b4fb000 in term 2.
I20240503 07:59:47.521561 10872 raft_consensus.cc:2781] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:47.521729 10872 raft_consensus.cc:686] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 2 LEADER]: Becoming Leader. State: Replica: ac6ca2734007406792d116f67b4fb000, State: Running, Role: LEADER
I20240503 07:59:47.522043 10872 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 537, Committed index: 537, Last appended: 1.537, Last appended by leader: 537, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:47.525627 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:47.527132 6738 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 reported cstate change: term changed from 1 to 2, leader changed from 76091450575f4076bd09550833d46f68 (127.0.99.130) to ac6ca2734007406792d116f67b4fb000 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "ac6ca2734007406792d116f67b4fb000" committed_config { opid_index: 462 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:47.529791 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:47.565225 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:47.568013 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 07:59:47.573210 6978 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44638: Illegal state: replica 76091450575f4076bd09550833d46f68 is not leader of this config: current role FOLLOWER
I20240503 07:59:47.573743 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:47.610083 8485 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46120: Illegal state: replica 5629c2d36c84404dbef6e9e66860cc8d is not leader of this config: current role FOLLOWER
I20240503 07:59:47.612691 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:47.630337 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:47.654881 7018 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Refusing update from remote peer ac6ca2734007406792d116f67b4fb000: Log matching property violated. Preceding OpId in replica: term: 1 index: 537. Preceding OpId from leader: term: 2 index: 539. (index mismatch)
I20240503 07:59:47.655189 8523 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Refusing update from remote peer ac6ca2734007406792d116f67b4fb000: Log matching property violated. Preceding OpId in replica: term: 1 index: 537. Preceding OpId from leader: term: 2 index: 539. (index mismatch)
I20240503 07:59:47.655650 10872 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 538, Last known committed idx: 537, Time since last communication: 0.000s
I20240503 07:59:47.656116 10876 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 538, Last known committed idx: 537, Time since last communication: 0.000s
I20240503 07:59:47.659243 10686 mvcc.cc:205] Tried to move back new op lower bound from 7023506176627859456 to 7023506176092659712. Current Snapshot: MvccSnapshot[applied={T|T < 7023506176627859456}]
I20240503 07:59:47.659516 10111 mvcc.cc:205] Tried to move back new op lower bound from 7023506176627859456 to 7023506176092659712. Current Snapshot: MvccSnapshot[applied={T|T < 7023506176627859456 or (T in {7023506176627859456})}]
I20240503 07:59:47.659914 10382 mvcc.cc:205] Tried to move back new op lower bound from 7023506176627859456 to 7023506176092659712. Current Snapshot: MvccSnapshot[applied={T|T < 7023506176627859456}]
I20240503 07:59:47.687266 7150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8"
dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45046
I20240503 07:59:47.687436 7150 raft_consensus.cc:595] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:47.715819 7109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.718843 7109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.734135 7109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.749416 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.757252 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.779603 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.799785 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.811944 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.822278 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
I20240503 07:59:47.825392 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:47.836874 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
I20240503 07:59:47.842388 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:47.852458 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 07:59:47.853722 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
I20240503 07:59:47.859961 7129 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:47.870536 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 07:59:47.871277 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.887830 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.902240 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
I20240503 07:59:47.905109 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:47.915089 8112 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:59:47.923862 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.944931 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.964073 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:47.985671 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.007066 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.029690 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.056371 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.082027 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.108866 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
W20240503 07:59:48.133237 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Service unavailable: leader transfer in progress
I20240503 07:59:48.152593 10651 raft_consensus.cc:979] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: : Instructing follower faca354acbbb49c89f3bbf9b3798eae6 to start an election
I20240503 07:59:48.152704 10651 raft_consensus.cc:1067] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Signalling peer faca354acbbb49c89f3bbf9b3798eae6 to start an election
I20240503 07:59:48.153313 8131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8"
dest_uuid: "faca354acbbb49c89f3bbf9b3798eae6"
from {username='slave'} at 127.0.99.131:43877
I20240503 07:59:48.153498 8131 raft_consensus.cc:482] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:48.153681 8131 raft_consensus.cc:3037] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:48.154801 8131 raft_consensus.cc:504] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:48.156522 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8" candidate_uuid: "faca354acbbb49c89f3bbf9b3798eae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 587 } } ignore_live_leader: true dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:48.156634 7150 raft_consensus.cc:3032] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:48.156677 7150 raft_consensus.cc:729] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:48.156822 7150 consensus_queue.cc:260] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 587, Committed index: 587, Last appended: 1.587, Last appended by leader: 587, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:48.156961 7150 raft_consensus.cc:3037] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:48.158900 7150 raft_consensus.cc:2454] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faca354acbbb49c89f3bbf9b3798eae6 in term 2.
I20240503 07:59:48.159647 8066 leader_election.cc:302] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79339180bc914f17a9f9f196b671f8ad, faca354acbbb49c89f3bbf9b3798eae6; no voters:
I20240503 07:59:48.160151 10957 raft_consensus.cc:2781] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:48.160300 10957 raft_consensus.cc:686] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 LEADER]: Becoming Leader. State: Replica: faca354acbbb49c89f3bbf9b3798eae6, State: Running, Role: LEADER
I20240503 07:59:48.160581 10957 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 587, Committed index: 587, Last appended: 1.587, Last appended by leader: 587, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:48.162526 6739 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 reported cstate change: term changed from 1 to 2, leader changed from 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) to faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "faca354acbbb49c89f3bbf9b3798eae6" committed_config { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
W20240503 07:59:48.163355 7111 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34632: Illegal state: replica 79339180bc914f17a9f9f196b671f8ad is not leader of this config: current role FOLLOWER
I20240503 07:59:48.164238 8131 leader_election.cc:288] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [CANDIDATE]: Term 2 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681), 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133)
I20240503 07:59:48.180341 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce487b8bb9694381923b19d4e48257c8" candidate_uuid: "faca354acbbb49c89f3bbf9b3798eae6" candidate_term: 2 candidate_status { last_received { term: 1 index: 587 } } ignore_live_leader: true dest_uuid: "6dc3aabcda15467498f2d0f01bbececf"
I20240503 07:59:48.180534 8261 raft_consensus.cc:3037] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:48.181664 8261 raft_consensus.cc:2454] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faca354acbbb49c89f3bbf9b3798eae6 in term 2.
I20240503 07:59:48.246724 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:48.252550 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:48.259552 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.267096 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.270988 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:48.276175 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:48.278573 7148 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Refusing update from remote peer faca354acbbb49c89f3bbf9b3798eae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 587. Preceding OpId from leader: term: 2 index: 589. (index mismatch)
I20240503 07:59:48.278584 8263 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Refusing update from remote peer faca354acbbb49c89f3bbf9b3798eae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 587. Preceding OpId from leader: term: 2 index: 589. (index mismatch)
I20240503 07:59:48.279095 10957 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 588, Last known committed idx: 587, Time since last communication: 0.000s
I20240503 07:59:48.279490 10964 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 588, Last known committed idx: 587, Time since last communication: 0.000s
I20240503 07:59:48.281482 10974 mvcc.cc:205] Tried to move back new op lower bound from 7023506179185221632 to 7023506178708471808. Current Snapshot: MvccSnapshot[applied={T|T < 7023506179185221632}]
I20240503 07:59:48.353590 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.504021 8394 consensus_queue.cc:237] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 570, Committed index: 570, Last appended: 2.570, Last appended by leader: 537, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } }
I20240503 07:59:48.505890 8522 raft_consensus.cc:1261] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Refusing update from remote peer ac6ca2734007406792d116f67b4fb000: Log matching property violated. Preceding OpId in replica: term: 2 index: 570. Preceding OpId from leader: term: 2 index: 571. (index mismatch)
I20240503 07:59:48.507761 10872 consensus_queue.cc:1035] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240503 07:59:48.509819 10926 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 [term 2 LEADER]: Committing config change with OpId 2.571: config changed from index 462 to 571, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:48.511176 8522 raft_consensus.cc:2932] T e21fe5b5f5a04168850a773226658450 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Committing config change with OpId 2.571: config changed from index 462 to 571, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:48.512301 6747 catalog_manager.cc:5484] T e21fe5b5f5a04168850a773226658450 P ac6ca2734007406792d116f67b4fb000 reported cstate change: config changed from index 462 to 571, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "ac6ca2734007406792d116f67b4fb000" committed_config { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } }
I20240503 07:59:48.519630 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet e21fe5b5f5a04168850a773226658450 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 571) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:48.524211 11014 tablet_replica.cc:330] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:48.525264 11014 raft_consensus.cc:2229] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:48.525478 11014 raft_consensus.cc:2258] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:48.526259 11014 ts_tablet_manager.cc:1875] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:48.535473 11014 ts_tablet_manager.cc:1888] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.570
I20240503 07:59:48.535632 11014 log.cc:1196] T e21fe5b5f5a04168850a773226658450 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e21fe5b5f5a04168850a773226658450
I20240503 07:59:48.536255 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet e21fe5b5f5a04168850a773226658450 (table rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]) successfully deleted
I20240503 07:59:48.755105 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:48.755358 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:48.784490 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.803061 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:48.803342 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.822707 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:48.846056 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:48.927702 8131 consensus_queue.cc:237] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 597, Committed index: 597, Last appended: 2.597, Last appended by leader: 587, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 598 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } }
I20240503 07:59:48.929152 8263 raft_consensus.cc:1261] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Refusing update from remote peer faca354acbbb49c89f3bbf9b3798eae6: Log matching property violated. Preceding OpId in replica: term: 2 index: 597. Preceding OpId from leader: term: 2 index: 598. (index mismatch)
I20240503 07:59:48.929962 10957 consensus_queue.cc:1035] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 598, Last known committed idx: 597, Time since last communication: 0.000s
I20240503 07:59:48.935598 10957 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 LEADER]: Committing config change with OpId 2.598: config changed from index 507 to 598, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 598 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:48.937930 6747 catalog_manager.cc:5484] T ce487b8bb9694381923b19d4e48257c8 P faca354acbbb49c89f3bbf9b3798eae6 reported cstate change: config changed from index 507 to 598, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "faca354acbbb49c89f3bbf9b3798eae6" committed_config { opid_index: 598 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:48.938179 8263 raft_consensus.cc:2932] T ce487b8bb9694381923b19d4e48257c8 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Committing config change with OpId 2.598: config changed from index 507 to 598, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 598 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } }
I20240503 07:59:48.940577 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet ce487b8bb9694381923b19d4e48257c8 with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 598) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:48.941478 11073 tablet_replica.cc:330] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:48.941648 11073 raft_consensus.cc:2229] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:48.941819 11073 raft_consensus.cc:2258] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:48.942730 11073 ts_tablet_manager.cc:1875] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:48.946435 11073 ts_tablet_manager.cc:1888] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.597
I20240503 07:59:48.946561 11073 log.cc:1196] T ce487b8bb9694381923b19d4e48257c8 P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:48.947773 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet ce487b8bb9694381923b19d4e48257c8 (table rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]) successfully deleted
I20240503 07:59:49.380828 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:49.395964 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.398108 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:49.403064 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.403334 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:49.411237 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:49.499838 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.749755 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:49.754190 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:49.755250 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.768802 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.797999 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:49.806142 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:49.815843 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:50.470217 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:50.476936 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:50.485103 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:50.499075 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:50.499403 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:50.536701 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:50.557914 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:50.835373 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:50.865898 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:50.868744 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:50.876909 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:50.876991 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:50.877712 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:50.976214 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:51.174481 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:51.175904 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:51.177277 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:51.179747 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:51.185321 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:51.187252 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:51.196247 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
8d366789da3a463e80dd995b3ff2311f | 127.0.99.190:43653 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35585 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5629c2d36c84404dbef6e9e66860cc8d | 127.0.99.135:35189 | HEALTHY | <none> | 0 | 0
625353768527407d8849faefd6527f2d | 127.0.99.129:43605 | HEALTHY | <none> | 5 | 0
6dc3aabcda15467498f2d0f01bbececf | 127.0.99.133:42133 | HEALTHY | <none> | 1 | 0
76091450575f4076bd09550833d46f68 | 127.0.99.130:43735 | HEALTHY | <none> | 5 | 0
79339180bc914f17a9f9f196b671f8ad | 127.0.99.131:38681 | HEALTHY | <none> | 3 | 0
ac6ca2734007406792d116f67b4fb000 | 127.0.99.134:34081 | HEALTHY | <none> | 1 | 0
faca354acbbb49c89f3bbf9b3798eae6 | 127.0.99.132:35347 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43605
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43735
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38681
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35347
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:42133
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:34081
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:35189
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43605
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43735
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38681
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35347
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:42133
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:34081
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:35189
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35585 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:59:51.225185 398 log_verifier.cc:126] Checking tablet 0cfcbed75e424ea3a337b51ba0bdd9e4
I20240503 07:59:51.339689 398 log_verifier.cc:177] Verified matching terms for 729 ops in tablet 0cfcbed75e424ea3a337b51ba0bdd9e4
I20240503 07:59:51.339934 398 log_verifier.cc:126] Checking tablet 61da585c42664c72b8624641b1dac5dd
I20240503 07:59:51.451308 398 log_verifier.cc:177] Verified matching terms for 733 ops in tablet 61da585c42664c72b8624641b1dac5dd
I20240503 07:59:51.451539 398 log_verifier.cc:126] Checking tablet 63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:51.572307 398 log_verifier.cc:177] Verified matching terms for 768 ops in tablet 63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:51.572517 398 log_verifier.cc:126] Checking tablet 73029d928463492abf2ba405ce28763c
I20240503 07:59:51.685180 398 log_verifier.cc:177] Verified matching terms for 709 ops in tablet 73029d928463492abf2ba405ce28763c
I20240503 07:59:51.685381 398 log_verifier.cc:126] Checking tablet 746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:51.796130 398 log_verifier.cc:177] Verified matching terms for 725 ops in tablet 746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:51.796358 398 log_verifier.cc:126] Checking tablet 7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:51.900611 398 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:51.900825 398 log_verifier.cc:126] Checking tablet 7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:52.011654 398 log_verifier.cc:177] Verified matching terms for 693 ops in tablet 7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:52.011885 398 log_verifier.cc:126] Checking tablet 829a4c6cc5d4441d907509c203635672
I20240503 07:59:52.124441 398 log_verifier.cc:177] Verified matching terms for 717 ops in tablet 829a4c6cc5d4441d907509c203635672
I20240503 07:59:52.124650 398 log_verifier.cc:126] Checking tablet 8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:52.140146 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:52.140206 398 log_verifier.cc:126] Checking tablet a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:52.252454 398 log_verifier.cc:177] Verified matching terms for 731 ops in tablet a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:52.252672 398 log_verifier.cc:126] Checking tablet a786879136e44b8d99311bec38c2f85d
I20240503 07:59:52.370061 398 log_verifier.cc:177] Verified matching terms for 773 ops in tablet a786879136e44b8d99311bec38c2f85d
I20240503 07:59:52.370296 398 log_verifier.cc:126] Checking tablet ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:52.477339 398 log_verifier.cc:177] Verified matching terms for 683 ops in tablet ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:52.477547 398 log_verifier.cc:126] Checking tablet ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:52.585107 398 log_verifier.cc:177] Verified matching terms for 683 ops in tablet ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:52.585314 398 log_verifier.cc:126] Checking tablet dfba18f2f9fe4d259a369ad071f8c274
I20240503 07:59:52.696887 398 log_verifier.cc:177] Verified matching terms for 723 ops in tablet dfba18f2f9fe4d259a369ad071f8c274
I20240503 07:59:52.697079 398 log_verifier.cc:126] Checking tablet e21fe5b5f5a04168850a773226658450
I20240503 07:59:52.798879 398 log_verifier.cc:177] Verified matching terms for 656 ops in tablet e21fe5b5f5a04168850a773226658450
I20240503 07:59:52.799118 398 log_verifier.cc:126] Checking tablet f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:52.902240 398 log_verifier.cc:177] Verified matching terms for 667 ops in tablet f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:52.904409 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--builtin_ntp_servers=127.0.99.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:53.012519 11302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.012732 11302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:53.012789 11302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.015825 11302 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:53.015888 11302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:53.015966 11302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:53.019546 11302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43653
--never_fsync=true
--heap_profile_path=/tmp/kudu.11302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:53.020424 11302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:53.021553 11302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.029724 11308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.030359 11307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.030912 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.031515 11302 server_base.cc:1034] running on GCE node
I20240503 07:59:53.031852 11302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:53.032478 11302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:53.033625 11302 hybrid_clock.cc:648] HybridClock initialized: now 1714723193033611 us; error 28 us; skew 500 ppm
I20240503 07:59:53.035446 11302 webserver.cc:460] Webserver started at http://127.0.99.136:33931/ using document root <none> and password file <none>
I20240503 07:59:53.035976 11302 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.036031 11302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.036192 11302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:53.037778 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "53c2e8878fff4f028714fc9bf31b770c"
format_stamp: "Formatted at 2024-05-03 07:59:53 on dist-test-slave-11df"
I20240503 07:59:53.038260 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "53c2e8878fff4f028714fc9bf31b770c"
format_stamp: "Formatted at 2024-05-03 07:59:53 on dist-test-slave-11df"
I20240503 07:59:53.041388 11302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:53.043342 11316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.044240 11302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:53.044337 11302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "53c2e8878fff4f028714fc9bf31b770c"
format_stamp: "Formatted at 2024-05-03 07:59:53 on dist-test-slave-11df"
I20240503 07:59:53.044435 11302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.059804 11302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.060349 11302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:53.060492 11302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.061044 11302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.061861 11302 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:53.061913 11302 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.061964 11302 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:53.061990 11302 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.070091 11302 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38647
I20240503 07:59:53.070145 11429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38647 every 8 connection(s)
I20240503 07:59:53.071062 11302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:53.079501 11430 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43653
I20240503 07:59:53.079659 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 11302
I20240503 07:59:53.079697 11430 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.079774 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:59:53.080104 11430 heartbeater.cc:511] Master 127.0.99.190:43653 requested a full tablet report, sending...
I20240503 07:59:53.080911 6738 ts_manager.cc:194] Registered new tserver with Master: 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136:38647)
I20240503 07:59:53.081511 6738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:40917
I20240503 07:59:53.272238 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.275882 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.278983 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.282039 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.282395 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.282728 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.302872 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.314474 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.391860 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.393532 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.396539 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.397764 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.400234 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.401752 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.428735 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.429328 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.465046 7150 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 731, Committed index: 731, Last appended: 1.731, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:53.466429 8132 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 731. Preceding OpId from leader: term: 1 index: 732. (index mismatch)
I20240503 07:59:53.466876 11536 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 732, Last known committed idx: 731, Time since last communication: 0.000s
I20240503 07:59:53.469074 8654 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.732: config changed from index 438 to 732, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.472121 8132 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Committing config change with OpId 1.732: config changed from index 438 to 732, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.472455 6744 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 438 to 732, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
W20240503 07:59:53.472697 7085 consensus_peers.cc:485] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad -> Peer 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136:38647): Couldn't send request to peer 53c2e8878fff4f028714fc9bf31b770c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a37812b3d0a346e3a1bc9c57956f2ca5. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:53.491070 7150 consensus_queue.cc:237] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 683, Committed index: 683, Last appended: 1.683, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
W20240503 07:59:53.492444 7085 consensus_peers.cc:485] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad -> Peer 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136:38647): Couldn't send request to peer 53c2e8878fff4f028714fc9bf31b770c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba2f6cdcc0704518a2931544ab969df8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:53.492484 8393 raft_consensus.cc:1261] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 683. Preceding OpId from leader: term: 1 index: 684. (index mismatch)
I20240503 07:59:53.492858 11536 consensus_queue.cc:1035] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 684, Last known committed idx: 683, Time since last communication: 0.000s
I20240503 07:59:53.494381 8654 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.684: config changed from index 372 to 684, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.494757 8393 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Committing config change with OpId 1.684: config changed from index 372 to 684, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.496883 6739 catalog_manager.cc:5484] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 372 to 684, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.512159 6888 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 717, Committed index: 717, Last appended: 1.717, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:53.513486 8261 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Refusing update from remote peer 625353768527407d8849faefd6527f2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 717. Preceding OpId from leader: term: 1 index: 718. (index mismatch)
I20240503 07:59:53.513852 11577 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 718, Last known committed idx: 717, Time since last communication: 0.000s
I20240503 07:59:53.515632 11194 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Committing config change with OpId 1.718: config changed from index 212 to 718, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
W20240503 07:59:53.516659 6823 consensus_peers.cc:485] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d -> Peer 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136:38647): Couldn't send request to peer 53c2e8878fff4f028714fc9bf31b770c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829a4c6cc5d4441d907509c203635672. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:53.517863 6747 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d reported cstate change: config changed from index 212 to 718, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "625353768527407d8849faefd6527f2d" committed_config { opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.519414 8261 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Committing config change with OpId 1.718: config changed from index 212 to 718, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.538251 7018 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 693, Committed index: 693, Last appended: 1.693, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:53.539563 8522 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Refusing update from remote peer 76091450575f4076bd09550833d46f68: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240503 07:59:53.539952 11596 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240503 07:59:53.541536 10417 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Committing config change with OpId 1.694: config changed from index 150 to 694, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
W20240503 07:59:53.542692 6954 consensus_peers.cc:485] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 -> Peer 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136:38647): Couldn't send request to peer 53c2e8878fff4f028714fc9bf31b770c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f94cbfb150547038338fbc0d892c8a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:53.543511 8522 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 150 to 694, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.543884 6739 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 reported cstate change: config changed from index 150 to 694, VOTER 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "76091450575f4076bd09550833d46f68" committed_config { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:53.601375 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.602864 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.605427 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.609752 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.611609 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.614581 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.614902 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.633191 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.701658 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.703195 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.707782 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.711213 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.714025 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.728284 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.729528 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.734185 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.813738 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.814648 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.817274 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.819101 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.822749 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.829504 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.832166 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.839056 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.885654 11720 ts_tablet_manager.cc:902] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:53.886335 11720 tablet_copy_client.cc:308] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:53.889886 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet a37812b3d0a346e3a1bc9c57956f2ca5 from peer 53c2e8878fff4f028714fc9bf31b770c ({username='slave'} at 127.0.99.136:33087)
I20240503 07:59:53.889986 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet a37812b3d0a346e3a1bc9c57956f2ca5 from peer 53c2e8878fff4f028714fc9bf31b770c at {username='slave'} at 127.0.99.136:33087: session id = 53c2e8878fff4f028714fc9bf31b770c-a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:53.890884 7171 tablet_copy_source_session.cc:205] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:53.893985 11720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37812b3d0a346e3a1bc9c57956f2ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.899058 11720 tablet_copy_client.cc:783] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:53.899327 11720 tablet_copy_client.cc:647] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:53.903303 11720 tablet_copy_client.cc:523] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:53.905134 11720 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap starting.
I20240503 07:59:53.927577 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.928656 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.932087 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:53.942479 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:53.945755 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.949064 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:53.955230 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.963083 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:53.998957 11720 log.cc:824] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:54.018756 11760 ts_tablet_manager.cc:902] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Initiating tablet copy from peer 625353768527407d8849faefd6527f2d (127.0.99.129:43605)
I20240503 07:59:54.019387 11760 tablet_copy_client.cc:308] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43605
I20240503 07:59:54.025933 6909 tablet_copy_service.cc:140] P 625353768527407d8849faefd6527f2d: Received BeginTabletCopySession request for tablet 829a4c6cc5d4441d907509c203635672 from peer 53c2e8878fff4f028714fc9bf31b770c ({username='slave'} at 127.0.99.136:48525)
I20240503 07:59:54.026043 6909 tablet_copy_service.cc:161] P 625353768527407d8849faefd6527f2d: Beginning new tablet copy session on tablet 829a4c6cc5d4441d907509c203635672 from peer 53c2e8878fff4f028714fc9bf31b770c at {username='slave'} at 127.0.99.136:48525: session id = 53c2e8878fff4f028714fc9bf31b770c-829a4c6cc5d4441d907509c203635672
I20240503 07:59:54.026820 6909 tablet_copy_source_session.cc:205] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:54.027662 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829a4c6cc5d4441d907509c203635672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:54.029706 11760 tablet_copy_client.cc:783] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:54.029850 11760 tablet_copy_client.cc:647] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:54.032938 11760 tablet_copy_client.cc:523] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:54.034193 11760 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap starting.
I20240503 07:59:54.077339 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.077440 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.083357 11430 heartbeater.cc:503] Master 127.0.99.190:43653 was elected leader, sending a full tablet report...
I20240503 07:59:54.084210 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.107039 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.108078 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.115257 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.115573 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.115808 11793 ts_tablet_manager.cc:902] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Initiating tablet copy from peer 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681)
I20240503 07:59:54.116984 11793 tablet_copy_client.cc:308] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38681
I20240503 07:59:54.120170 7171 tablet_copy_service.cc:140] P 79339180bc914f17a9f9f196b671f8ad: Received BeginTabletCopySession request for tablet ba2f6cdcc0704518a2931544ab969df8 from peer 53c2e8878fff4f028714fc9bf31b770c ({username='slave'} at 127.0.99.136:33087)
I20240503 07:59:54.120283 7171 tablet_copy_service.cc:161] P 79339180bc914f17a9f9f196b671f8ad: Beginning new tablet copy session on tablet ba2f6cdcc0704518a2931544ab969df8 from peer 53c2e8878fff4f028714fc9bf31b770c at {username='slave'} at 127.0.99.136:33087: session id = 53c2e8878fff4f028714fc9bf31b770c-ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:54.120301 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.121388 7171 tablet_copy_source_session.cc:205] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:54.123538 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2f6cdcc0704518a2931544ab969df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:54.126020 11793 tablet_copy_client.cc:783] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:54.126202 11793 tablet_copy_client.cc:647] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:54.135965 11795 ts_tablet_manager.cc:902] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Initiating tablet copy from peer 76091450575f4076bd09550833d46f68 (127.0.99.130:43735)
I20240503 07:59:54.136575 11795 tablet_copy_client.cc:308] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43735
I20240503 07:59:54.141405 7039 tablet_copy_service.cc:140] P 76091450575f4076bd09550833d46f68: Received BeginTabletCopySession request for tablet 7f94cbfb150547038338fbc0d892c8a6 from peer 53c2e8878fff4f028714fc9bf31b770c ({username='slave'} at 127.0.99.136:40369)
I20240503 07:59:54.141530 7039 tablet_copy_service.cc:161] P 76091450575f4076bd09550833d46f68: Beginning new tablet copy session on tablet 7f94cbfb150547038338fbc0d892c8a6 from peer 53c2e8878fff4f028714fc9bf31b770c at {username='slave'} at 127.0.99.136:40369: session id = 53c2e8878fff4f028714fc9bf31b770c-7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:54.142411 7039 tablet_copy_source_session.cc:205] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:54.143352 11795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f94cbfb150547038338fbc0d892c8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:54.145790 11795 tablet_copy_client.cc:783] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:54.145970 11795 tablet_copy_client.cc:647] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:54.146976 11793 tablet_copy_client.cc:523] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:54.148154 11793 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap starting.
I20240503 07:59:54.148517 11795 tablet_copy_client.cc:523] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:54.150997 11795 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap starting.
I20240503 07:59:54.293872 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.294404 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.294840 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.313771 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.327692 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.355094 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.373457 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.385282 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.536176 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.537971 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.551021 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.565824 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.568701 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.576680 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.606057 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.611119 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.679467 11720 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap replayed 1/1 log segments. Stats: ops{read=732 overwritten=0 applied=732 ignored=0} inserts{seen=1415 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:54.680049 11720 tablet_bootstrap.cc:492] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap complete.
I20240503 07:59:54.680574 11720 ts_tablet_manager.cc:1367] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Time spent bootstrapping tablet: real 0.776s user 0.482s sys 0.038s
I20240503 07:59:54.683359 11720 raft_consensus.cc:348] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.683725 11720 raft_consensus.cc:729] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Initialized, Role: FOLLOWER
I20240503 07:59:54.684280 11720 consensus_queue.cc:260] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 732, Last appended: 1.732, Last appended by leader: 732, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.685266 11720 ts_tablet_manager.cc:1398] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:54.688414 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session 53c2e8878fff4f028714fc9bf31b770c-a37812b3d0a346e3a1bc9c57956f2ca5 received from {username='slave'} at 127.0.99.136:33087
I20240503 07:59:54.688516 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session 53c2e8878fff4f028714fc9bf31b770c-a37812b3d0a346e3a1bc9c57956f2ca5 on tablet a37812b3d0a346e3a1bc9c57956f2ca5 with peer 53c2e8878fff4f028714fc9bf31b770c
I20240503 07:59:54.731810 11384 raft_consensus.cc:1203] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.731->[1.732-1.732] Dedup: 1.732->[]
I20240503 07:59:54.737639 11760 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap replayed 1/1 log segments. Stats: ops{read=718 overwritten=0 applied=718 ignored=0} inserts{seen=1401 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:54.738003 11760 tablet_bootstrap.cc:492] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap complete.
I20240503 07:59:54.738267 11760 ts_tablet_manager.cc:1367] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Time spent bootstrapping tablet: real 0.704s user 0.468s sys 0.027s
I20240503 07:59:54.738847 11760 raft_consensus.cc:348] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.739032 11760 raft_consensus.cc:729] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Initialized, Role: FOLLOWER
I20240503 07:59:54.739171 11760 consensus_queue.cc:260] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 718, Last appended: 1.718, Last appended by leader: 718, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.739499 11760 ts_tablet_manager.cc:1398] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:54.740191 6909 tablet_copy_service.cc:342] P 625353768527407d8849faefd6527f2d: Request end of tablet copy session 53c2e8878fff4f028714fc9bf31b770c-829a4c6cc5d4441d907509c203635672 received from {username='slave'} at 127.0.99.136:48525
I20240503 07:59:54.740270 6909 tablet_copy_service.cc:433] P 625353768527407d8849faefd6527f2d: ending tablet copy session 53c2e8878fff4f028714fc9bf31b770c-829a4c6cc5d4441d907509c203635672 on tablet 829a4c6cc5d4441d907509c203635672 with peer 53c2e8878fff4f028714fc9bf31b770c
I20240503 07:59:54.792541 11793 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap replayed 1/1 log segments. Stats: ops{read=684 overwritten=0 applied=684 ignored=0} inserts{seen=1367 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:54.792913 11793 tablet_bootstrap.cc:492] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap complete.
I20240503 07:59:54.793167 11793 ts_tablet_manager.cc:1367] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Time spent bootstrapping tablet: real 0.645s user 0.458s sys 0.029s
I20240503 07:59:54.793687 11793 raft_consensus.cc:348] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.793817 11793 raft_consensus.cc:729] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Initialized, Role: FOLLOWER
I20240503 07:59:54.793970 11793 consensus_queue.cc:260] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 684, Last appended: 1.684, Last appended by leader: 684, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 684 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "ac6ca2734007406792d116f67b4fb000" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34081 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.794265 11793 ts_tablet_manager.cc:1398] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:54.794961 7171 tablet_copy_service.cc:342] P 79339180bc914f17a9f9f196b671f8ad: Request end of tablet copy session 53c2e8878fff4f028714fc9bf31b770c-ba2f6cdcc0704518a2931544ab969df8 received from {username='slave'} at 127.0.99.136:33087
I20240503 07:59:54.795054 7171 tablet_copy_service.cc:433] P 79339180bc914f17a9f9f196b671f8ad: ending tablet copy session 53c2e8878fff4f028714fc9bf31b770c-ba2f6cdcc0704518a2931544ab969df8 on tablet ba2f6cdcc0704518a2931544ab969df8 with peer 53c2e8878fff4f028714fc9bf31b770c
I20240503 07:59:54.867756 11384 raft_consensus.cc:1203] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.717->[1.718-1.718] Dedup: 1.718->[]
I20240503 07:59:54.902297 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.903523 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.905076 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.906527 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.912020 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.921092 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.921772 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.936259 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:54.959558 11795 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap replayed 1/1 log segments. Stats: ops{read=694 overwritten=0 applied=694 ignored=0} inserts{seen=1323 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:54.959916 11795 tablet_bootstrap.cc:492] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Bootstrap complete.
I20240503 07:59:54.960180 11795 ts_tablet_manager.cc:1367] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Time spent bootstrapping tablet: real 0.809s user 0.441s sys 0.036s
I20240503 07:59:54.960701 11795 raft_consensus.cc:348] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.960842 11795 raft_consensus.cc:729] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Initialized, Role: FOLLOWER
I20240503 07:59:54.961030 11795 consensus_queue.cc:260] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:54.961799 11795 ts_tablet_manager.cc:1398] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:54.963126 7039 tablet_copy_service.cc:342] P 76091450575f4076bd09550833d46f68: Request end of tablet copy session 53c2e8878fff4f028714fc9bf31b770c-7f94cbfb150547038338fbc0d892c8a6 received from {username='slave'} at 127.0.99.136:40369
I20240503 07:59:54.963214 7039 tablet_copy_service.cc:433] P 76091450575f4076bd09550833d46f68: ending tablet copy session 53c2e8878fff4f028714fc9bf31b770c-7f94cbfb150547038338fbc0d892c8a6 on tablet 7f94cbfb150547038338fbc0d892c8a6 with peer 53c2e8878fff4f028714fc9bf31b770c
I20240503 07:59:55.033061 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.034054 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.035223 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.035264 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.035573 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.036680 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:55.043025 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.047196 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.148141 11384 raft_consensus.cc:1203] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.683->[1.684-1.684] Dedup: 1.684->[]
I20240503 07:59:55.161798 11364 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.163920 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.164834 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.165378 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.166374 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.178889 7129 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:55.180047 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.199050 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.212990 11384 raft_consensus.cc:1203] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.693->[1.694-1.694] Dedup: 1.694->[]
I20240503 07:59:55.230960 7150 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5"
dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51374
I20240503 07:59:55.231108 7150 raft_consensus.cc:595] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:55.234721 12003 raft_consensus.cc:979] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: : Instructing follower 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:55.234825 12003 raft_consensus.cc:1067] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Signalling peer 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:55.235299 11384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5"
dest_uuid: "53c2e8878fff4f028714fc9bf31b770c"
from {username='slave'} at 127.0.99.131:53629
I20240503 07:59:55.235498 11384 raft_consensus.cc:482] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:55.235623 11384 raft_consensus.cc:3037] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.236617 11384 raft_consensus.cc:504] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.237587 11384 leader_election.cc:288] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Requested vote from peers 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681), faca354acbbb49c89f3bbf9b3798eae6 (127.0.99.132:35347)
I20240503 07:59:55.238343 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 732 } } ignore_live_leader: true dest_uuid: "79339180bc914f17a9f9f196b671f8ad"
I20240503 07:59:55.238461 7150 raft_consensus.cc:3032] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:55.238500 7150 raft_consensus.cc:729] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79339180bc914f17a9f9f196b671f8ad, State: Running, Role: LEADER
I20240503 07:59:55.238632 7150 consensus_queue.cc:260] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 732, Committed index: 732, Last appended: 1.732, Last appended by leader: 732, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.238797 7150 raft_consensus.cc:3037] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.239892 7150 raft_consensus.cc:2454] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:55.240288 11318 leader_election.cc:302] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c2e8878fff4f028714fc9bf31b770c, 79339180bc914f17a9f9f196b671f8ad; no voters:
I20240503 07:59:55.240548 11880 raft_consensus.cc:2781] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:55.240618 11880 raft_consensus.cc:686] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Becoming Leader. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Running, Role: LEADER
I20240503 07:59:55.240943 11880 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 732, Committed index: 732, Last appended: 1.732, Last appended by leader: 732, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.243108 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37812b3d0a346e3a1bc9c57956f2ca5" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 732 } } ignore_live_leader: true dest_uuid: "faca354acbbb49c89f3bbf9b3798eae6"
I20240503 07:59:55.243270 8132 raft_consensus.cc:3037] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.244221 8132 raft_consensus.cc:2454] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:55.245574 6738 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c reported cstate change: term changed from 1 to 2, leader changed from 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) to 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136). New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 732 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:55.285647 11364 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.289091 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.293444 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.293783 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.293898 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.295545 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.301575 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.316358 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.394706 6868 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.395078 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.401226 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.403596 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.403975 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.405369 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.408648 11364 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.424234 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.452004 6888 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "829a4c6cc5d4441d907509c203635672"
dest_uuid: "625353768527407d8849faefd6527f2d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43116
I20240503 07:59:55.452147 6888 raft_consensus.cc:595] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:55.481621 12091 raft_consensus.cc:979] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: : Instructing follower 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:55.481782 11194 raft_consensus.cc:1067] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Signalling peer 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:55.482321 11384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "829a4c6cc5d4441d907509c203635672"
dest_uuid: "53c2e8878fff4f028714fc9bf31b770c"
from {username='slave'} at 127.0.99.129:45065
I20240503 07:59:55.482452 11384 raft_consensus.cc:482] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:55.482515 11384 raft_consensus.cc:3037] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.483410 11384 raft_consensus.cc:504] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.483824 11384 leader_election.cc:288] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Requested vote from peers 625353768527407d8849faefd6527f2d (127.0.99.129:43605), 6dc3aabcda15467498f2d0f01bbececf (127.0.99.133:42133)
I20240503 07:59:55.484206 6888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829a4c6cc5d4441d907509c203635672" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 718 } } ignore_live_leader: true dest_uuid: "625353768527407d8849faefd6527f2d"
I20240503 07:59:55.484329 6888 raft_consensus.cc:3032] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:55.484371 6888 raft_consensus.cc:729] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625353768527407d8849faefd6527f2d, State: Running, Role: LEADER
I20240503 07:59:55.484531 6888 consensus_queue.cc:260] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 718, Committed index: 718, Last appended: 1.718, Last appended by leader: 718, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.484701 6888 raft_consensus.cc:3037] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.485752 6888 raft_consensus.cc:2454] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:55.486066 11320 leader_election.cc:302] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c2e8878fff4f028714fc9bf31b770c, 625353768527407d8849faefd6527f2d; no voters:
I20240503 07:59:55.486243 11879 raft_consensus.cc:2781] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:55.486371 11879 raft_consensus.cc:686] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Becoming Leader. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Running, Role: LEADER
I20240503 07:59:55.486505 11879 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 718, Committed index: 718, Last appended: 1.718, Last appended by leader: 718, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.488116 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829a4c6cc5d4441d907509c203635672" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 718 } } ignore_live_leader: true dest_uuid: "6dc3aabcda15467498f2d0f01bbececf"
I20240503 07:59:55.488222 8261 raft_consensus.cc:3037] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.488153 6739 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c reported cstate change: term changed from 1 to 2, leader changed from 625353768527407d8849faefd6527f2d (127.0.99.129) to 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136). New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 718 OBSOLETE_local: false peers { permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 } } peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:55.489096 8261 raft_consensus.cc:2454] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:55.509656 11364 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.517266 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.518590 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:55.520808 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.521042 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.521063 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.531934 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.560557 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.631927 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.633767 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.633970 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.634711 11364 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.636039 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:55.640218 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.662072 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.662355 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.697188 7150 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 732. Preceding OpId from leader: term: 2 index: 733. (index mismatch)
I20240503 07:59:55.697660 11879 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 733, Last known committed idx: 732, Time since last communication: 0.000s
I20240503 07:59:55.700887 8132 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 732. Preceding OpId from leader: term: 2 index: 733. (index mismatch)
I20240503 07:59:55.701275 11880 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 733, Last known committed idx: 732, Time since last communication: 0.000s
I20240503 07:59:55.703498 7150 consensus_queue.cc:237] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 684, Committed index: 684, Last appended: 1.684, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 685 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:55.704835 11384 raft_consensus.cc:1261] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Refusing update from remote peer 79339180bc914f17a9f9f196b671f8ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 684. Preceding OpId from leader: term: 1 index: 685. (index mismatch)
I20240503 07:59:55.705222 12003 consensus_queue.cc:1035] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 685, Last known committed idx: 684, Time since last communication: 0.000s
I20240503 07:59:55.706807 11536 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad [term 1 LEADER]: Committing config change with OpId 1.685: config changed from index 684 to 685, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) evicted. New config: { opid_index: 685 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:55.707257 11384 raft_consensus.cc:2932] T ba2f6cdcc0704518a2931544ab969df8 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Committing config change with OpId 1.685: config changed from index 684 to 685, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) evicted. New config: { opid_index: 685 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:55.709185 6738 catalog_manager.cc:5484] T ba2f6cdcc0704518a2931544ab969df8 P 79339180bc914f17a9f9f196b671f8ad reported cstate change: config changed from index 684 to 685, VOTER ac6ca2734007406792d116f67b4fb000 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "79339180bc914f17a9f9f196b671f8ad" committed_config { opid_index: 685 OBSOLETE_local: false peers { permanent_uuid: "79339180bc914f17a9f9f196b671f8ad" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38681 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:55.714565 8374 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2f6cdcc0704518a2931544ab969df8 with delete_type TABLET_DATA_TOMBSTONED (TS ac6ca2734007406792d116f67b4fb000 not found in new config with opid_index 685) from {username='slave'} at 127.0.0.1:42148
I20240503 07:59:55.715013 12177 tablet_replica.cc:330] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: stopping tablet replica
I20240503 07:59:55.715281 12177 raft_consensus.cc:2229] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:55.715458 12177 raft_consensus.cc:2258] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:55.716760 12177 ts_tablet_manager.cc:1875] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:55.718502 12177 ts_tablet_manager.cc:1888] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.684
I20240503 07:59:55.718576 12177 log.cc:1196] T ba2f6cdcc0704518a2931544ab969df8 P ac6ca2734007406792d116f67b4fb000: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:55.719105 6733 catalog_manager.cc:4830] TS ac6ca2734007406792d116f67b4fb000 (127.0.99.134:34081): tablet ba2f6cdcc0704518a2931544ab969df8 (table rebalance_test_table_4 [id=e19cd3a96f5749c48f2a9cf4c183bee9]) successfully deleted
I20240503 07:59:55.971721 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.974264 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.974354 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:55.977051 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:55.978503 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:55.984668 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:55.989595 11364 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.002127 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.021795 8261 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 718. Preceding OpId from leader: term: 2 index: 719. (index mismatch)
I20240503 07:59:56.022213 11880 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 719, Last known committed idx: 718, Time since last communication: 0.000s
I20240503 07:59:56.024684 6888 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 718. Preceding OpId from leader: term: 2 index: 719. (index mismatch)
I20240503 07:59:56.025274 11880 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "625353768527407d8849faefd6527f2d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 719, Last known committed idx: 718, Time since last communication: 0.000s
I20240503 07:59:56.037544 7018 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6"
dest_uuid: "76091450575f4076bd09550833d46f68"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59532
I20240503 07:59:56.037688 7018 raft_consensus.cc:595] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:56.089290 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.092471 6998 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:56.099462 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.103233 11364 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.105008 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.106441 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.112999 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.145103 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.191812 11596 raft_consensus.cc:979] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: : Instructing follower 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:56.191916 11596 raft_consensus.cc:1067] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Signalling peer 53c2e8878fff4f028714fc9bf31b770c to start an election
I20240503 07:59:56.192413 11384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6"
dest_uuid: "53c2e8878fff4f028714fc9bf31b770c"
from {username='slave'} at 127.0.99.130:49021
I20240503 07:59:56.192584 11384 raft_consensus.cc:482] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:56.192644 11384 raft_consensus.cc:3037] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.193614 11384 raft_consensus.cc:504] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:56.194133 11384 leader_election.cc:288] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Requested vote from peers 76091450575f4076bd09550833d46f68 (127.0.99.130:43735), 5629c2d36c84404dbef6e9e66860cc8d (127.0.99.135:35189)
I20240503 07:59:56.194543 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: true dest_uuid: "76091450575f4076bd09550833d46f68"
I20240503 07:59:56.194661 7018 raft_consensus.cc:3032] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:56.194706 7018 raft_consensus.cc:729] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76091450575f4076bd09550833d46f68, State: Running, Role: LEADER
I20240503 07:59:56.194839 7018 consensus_queue.cc:260] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 694, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:56.195000 7018 raft_consensus.cc:3037] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.196017 7018 raft_consensus.cc:2454] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:56.196374 11318 leader_election.cc:302] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c2e8878fff4f028714fc9bf31b770c, 76091450575f4076bd09550833d46f68; no voters:
I20240503 07:59:56.196583 11880 raft_consensus.cc:2781] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:56.196681 11880 raft_consensus.cc:686] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Becoming Leader. State: Replica: 53c2e8878fff4f028714fc9bf31b770c, State: Running, Role: LEADER
I20240503 07:59:56.196813 11880 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 694, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:56.198592 6747 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c reported cstate change: term changed from 1 to 2, leader changed from 76091450575f4076bd09550833d46f68 (127.0.99.130) to 53c2e8878fff4f028714fc9bf31b770c (127.0.99.136). New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 } } peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.199446 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f94cbfb150547038338fbc0d892c8a6" candidate_uuid: "53c2e8878fff4f028714fc9bf31b770c" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: true dest_uuid: "5629c2d36c84404dbef6e9e66860cc8d"
I20240503 07:59:56.199579 8522 raft_consensus.cc:3037] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.200604 8522 raft_consensus.cc:2454] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c2e8878fff4f028714fc9bf31b770c in term 2.
I20240503 07:59:56.222142 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.222426 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:56.222431 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.232333 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.239675 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.239991 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.241004 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.242277 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.293165 11384 consensus_queue.cc:237] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 733, Committed index: 733, Last appended: 2.733, Last appended by leader: 732, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 734 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:56.294183 8132 raft_consensus.cc:1261] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 2 index: 733. Preceding OpId from leader: term: 2 index: 734. (index mismatch)
I20240503 07:59:56.294533 11879 consensus_queue.cc:1035] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 734, Last known committed idx: 733, Time since last communication: 0.000s
I20240503 07:59:56.296033 11880 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Committing config change with OpId 2.734: config changed from index 732 to 734, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 734 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.296463 8132 raft_consensus.cc:2932] T a37812b3d0a346e3a1bc9c57956f2ca5 P faca354acbbb49c89f3bbf9b3798eae6 [term 2 FOLLOWER]: Committing config change with OpId 2.734: config changed from index 732 to 734, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New config: { opid_index: 734 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.298861 6738 catalog_manager.cc:5484] T a37812b3d0a346e3a1bc9c57956f2ca5 P 53c2e8878fff4f028714fc9bf31b770c reported cstate change: config changed from index 732 to 734, VOTER 79339180bc914f17a9f9f196b671f8ad (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 734 OBSOLETE_local: false peers { permanent_uuid: "faca354acbbb49c89f3bbf9b3798eae6" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35347 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.301498 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet a37812b3d0a346e3a1bc9c57956f2ca5 with delete_type TABLET_DATA_TOMBSTONED (TS 79339180bc914f17a9f9f196b671f8ad not found in new config with opid_index 734) from {username='slave'} at 127.0.0.1:34554
I20240503 07:59:56.301802 12320 tablet_replica.cc:330] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: stopping tablet replica
I20240503 07:59:56.301949 12320 raft_consensus.cc:2229] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.302059 12320 raft_consensus.cc:2258] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.303506 12320 ts_tablet_manager.cc:1875] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.305753 12320 ts_tablet_manager.cc:1888] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.733
I20240503 07:59:56.305824 12320 log.cc:1196] T a37812b3d0a346e3a1bc9c57956f2ca5 P 79339180bc914f17a9f9f196b671f8ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:56.306389 6732 catalog_manager.cc:4830] TS 79339180bc914f17a9f9f196b671f8ad (127.0.99.131:38681): tablet a37812b3d0a346e3a1bc9c57956f2ca5 (table rebalance_test_table_2 [id=0460bdfa9f984a05aa7e341311792c91]) successfully deleted
I20240503 07:59:56.354274 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.358834 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:56.361023 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.363061 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.373507 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.386013 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.386662 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.388513 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.431615 11384 consensus_queue.cc:237] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 719, Committed index: 719, Last appended: 2.719, Last appended by leader: 718, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 720 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:56.432874 8261 raft_consensus.cc:1261] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 2 index: 719. Preceding OpId from leader: term: 2 index: 720. (index mismatch)
I20240503 07:59:56.433284 11880 consensus_queue.cc:1035] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 720, Last known committed idx: 719, Time since last communication: 0.000s
I20240503 07:59:56.435017 11880 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Committing config change with OpId 2.720: config changed from index 718 to 720, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 720 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.435369 8261 raft_consensus.cc:2932] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf [term 2 FOLLOWER]: Committing config change with OpId 2.720: config changed from index 718 to 720, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New config: { opid_index: 720 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.437824 6738 catalog_manager.cc:5484] T 829a4c6cc5d4441d907509c203635672 P 6dc3aabcda15467498f2d0f01bbececf reported cstate change: config changed from index 718 to 720, VOTER 625353768527407d8849faefd6527f2d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 720 OBSOLETE_local: false peers { permanent_uuid: "6dc3aabcda15467498f2d0f01bbececf" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42133 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:56.440655 6868 tablet_service.cc:1514] Processing DeleteTablet for tablet 829a4c6cc5d4441d907509c203635672 with delete_type TABLET_DATA_TOMBSTONED (TS 625353768527407d8849faefd6527f2d not found in new config with opid_index 720) from {username='slave'} at 127.0.0.1:54846
I20240503 07:59:56.441051 12374 tablet_replica.cc:330] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: stopping tablet replica
I20240503 07:59:56.441231 12374 raft_consensus.cc:2229] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.441358 12374 raft_consensus.cc:2258] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.442682 12374 ts_tablet_manager.cc:1875] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.445123 12374 ts_tablet_manager.cc:1888] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.719
I20240503 07:59:56.445209 12374 log.cc:1196] T 829a4c6cc5d4441d907509c203635672 P 625353768527407d8849faefd6527f2d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/829a4c6cc5d4441d907509c203635672
I20240503 07:59:56.445945 6734 catalog_manager.cc:4830] TS 625353768527407d8849faefd6527f2d (127.0.99.129:43605): tablet 829a4c6cc5d4441d907509c203635672 (table rebalance_test_table_3 [id=35a4ffdad9584bba8b9b1c48480a7dd7]) successfully deleted
I20240503 07:59:56.501904 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:56.503831 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.505069 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.507230 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.507930 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.510228 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.529171 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.541801 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.639611 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.641490 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.641566 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:56.642952 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.648099 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.648311 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.648404 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.674890 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.712390 7018 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 2 index: 695. (index mismatch)
I20240503 07:59:56.712839 11879 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76091450575f4076bd09550833d46f68" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240503 07:59:56.715335 8522 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 2 index: 695. (index mismatch)
I20240503 07:59:56.715673 11880 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240503 07:59:56.949939 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.951807 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.954735 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:56.960206 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.967061 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:56.967936 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:56.970741 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:56.973491 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.025038 11384 consensus_queue.cc:237] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 695, Committed index: 695, Last appended: 2.695, Last appended by leader: 694, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } }
I20240503 07:59:57.026167 8522 raft_consensus.cc:1261] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Refusing update from remote peer 53c2e8878fff4f028714fc9bf31b770c: Log matching property violated. Preceding OpId in replica: term: 2 index: 695. Preceding OpId from leader: term: 2 index: 696. (index mismatch)
I20240503 07:59:57.026561 11879 consensus_queue.cc:1035] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 696, Last known committed idx: 695, Time since last communication: 0.000s
I20240503 07:59:57.027997 11880 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c [term 2 LEADER]: Committing config change with OpId 2.696: config changed from index 694 to 696, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:57.028410 8522 raft_consensus.cc:2932] T 7f94cbfb150547038338fbc0d892c8a6 P 5629c2d36c84404dbef6e9e66860cc8d [term 2 FOLLOWER]: Committing config change with OpId 2.696: config changed from index 694 to 696, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New config: { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:57.030674 6738 catalog_manager.cc:5484] T 7f94cbfb150547038338fbc0d892c8a6 P 53c2e8878fff4f028714fc9bf31b770c reported cstate change: config changed from index 694 to 696, VOTER 76091450575f4076bd09550833d46f68 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "53c2e8878fff4f028714fc9bf31b770c" committed_config { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "5629c2d36c84404dbef6e9e66860cc8d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35189 } } peers { permanent_uuid: "53c2e8878fff4f028714fc9bf31b770c" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38647 } } }
I20240503 07:59:57.033396 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f94cbfb150547038338fbc0d892c8a6 with delete_type TABLET_DATA_TOMBSTONED (TS 76091450575f4076bd09550833d46f68 not found in new config with opid_index 696) from {username='slave'} at 127.0.0.1:44520
I20240503 07:59:57.033712 12523 tablet_replica.cc:330] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: stopping tablet replica
I20240503 07:59:57.033833 12523 raft_consensus.cc:2229] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:57.033928 12523 raft_consensus.cc:2258] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:57.035068 12523 ts_tablet_manager.cc:1875] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:57.036868 12523 ts_tablet_manager.cc:1888] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.695
I20240503 07:59:57.036928 12523 log.cc:1196] T 7f94cbfb150547038338fbc0d892c8a6 P 76091450575f4076bd09550833d46f68: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:57.037436 6732 catalog_manager.cc:4830] TS 76091450575f4076bd09550833d46f68 (127.0.99.130:43735): tablet 7f94cbfb150547038338fbc0d892c8a6 (table rebalance_test_table_0 [id=1b581afb82b0405c935a7bad8592c109]) successfully deleted
I20240503 07:59:57.291203 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:57.295944 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.295933 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.298568 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.306777 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.315798 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.321930 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.322872 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:57.402859 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:57.409394 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.412094 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.415197 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:57.415920 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.423501 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.426527 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.441984 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.528950 8505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.533950 11364 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:57.535209 8239 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.535693 8374 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.537331 6998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.551739 7129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:57.551884 6868 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.575795 8112 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
8d366789da3a463e80dd995b3ff2311f | 127.0.99.190:43653 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35585 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
53c2e8878fff4f028714fc9bf31b770c | 127.0.99.136:38647 | HEALTHY | <none> | 3 | 0
5629c2d36c84404dbef6e9e66860cc8d | 127.0.99.135:35189 | HEALTHY | <none> | 0 | 0
625353768527407d8849faefd6527f2d | 127.0.99.129:43605 | HEALTHY | <none> | 4 | 0
6dc3aabcda15467498f2d0f01bbececf | 127.0.99.133:42133 | HEALTHY | <none> | 1 | 0
76091450575f4076bd09550833d46f68 | 127.0.99.130:43735 | HEALTHY | <none> | 4 | 0
79339180bc914f17a9f9f196b671f8ad | 127.0.99.131:38681 | HEALTHY | <none> | 2 | 0
ac6ca2734007406792d116f67b4fb000 | 127.0.99.134:34081 | HEALTHY | <none> | 1 | 0
faca354acbbb49c89f3bbf9b3798eae6 | 127.0.99.132:35347 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43605
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43735
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38681
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35347
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:42133
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:34081
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:35189
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:38647
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43605
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43735
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38681
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35347
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:42133
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:34081
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:35189
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:38647
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35585 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:59:57.595732 398 log_verifier.cc:126] Checking tablet 0cfcbed75e424ea3a337b51ba0bdd9e4
I20240503 07:59:57.711113 398 log_verifier.cc:177] Verified matching terms for 729 ops in tablet 0cfcbed75e424ea3a337b51ba0bdd9e4
I20240503 07:59:57.711330 398 log_verifier.cc:126] Checking tablet 61da585c42664c72b8624641b1dac5dd
I20240503 07:59:57.821457 398 log_verifier.cc:177] Verified matching terms for 733 ops in tablet 61da585c42664c72b8624641b1dac5dd
I20240503 07:59:57.821682 398 log_verifier.cc:126] Checking tablet 63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:57.930996 398 log_verifier.cc:177] Verified matching terms for 768 ops in tablet 63ed07cc90364f058eb5ded00d3f7514
I20240503 07:59:57.931233 398 log_verifier.cc:126] Checking tablet 73029d928463492abf2ba405ce28763c
I20240503 07:59:58.035691 398 log_verifier.cc:177] Verified matching terms for 709 ops in tablet 73029d928463492abf2ba405ce28763c
I20240503 07:59:58.035910 398 log_verifier.cc:126] Checking tablet 746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:58.146795 398 log_verifier.cc:177] Verified matching terms for 725 ops in tablet 746df8ea7adb4b8cbd2b1782762683e2
I20240503 07:59:58.147032 398 log_verifier.cc:126] Checking tablet 7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:58.245965 398 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 7f2f2ab1fe39408e8ee235acee496da7
I20240503 07:59:58.246192 398 log_verifier.cc:126] Checking tablet 7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:58.354249 398 log_verifier.cc:177] Verified matching terms for 696 ops in tablet 7f94cbfb150547038338fbc0d892c8a6
I20240503 07:59:58.354478 398 log_verifier.cc:126] Checking tablet 829a4c6cc5d4441d907509c203635672
I20240503 07:59:58.461941 398 log_verifier.cc:177] Verified matching terms for 720 ops in tablet 829a4c6cc5d4441d907509c203635672
I20240503 07:59:58.462170 398 log_verifier.cc:126] Checking tablet 8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:58.471089 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8381a34bdbd84c9ca7dbe8d5aca087f2
I20240503 07:59:58.471153 398 log_verifier.cc:126] Checking tablet a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:58.584419 398 log_verifier.cc:177] Verified matching terms for 734 ops in tablet a37812b3d0a346e3a1bc9c57956f2ca5
I20240503 07:59:58.584631 398 log_verifier.cc:126] Checking tablet a786879136e44b8d99311bec38c2f85d
I20240503 07:59:58.700786 398 log_verifier.cc:177] Verified matching terms for 773 ops in tablet a786879136e44b8d99311bec38c2f85d
I20240503 07:59:58.701081 398 log_verifier.cc:126] Checking tablet ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:58.808526 398 log_verifier.cc:177] Verified matching terms for 685 ops in tablet ba2f6cdcc0704518a2931544ab969df8
I20240503 07:59:58.808732 398 log_verifier.cc:126] Checking tablet ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:58.908668 398 log_verifier.cc:177] Verified matching terms for 683 ops in tablet ce487b8bb9694381923b19d4e48257c8
I20240503 07:59:58.908891 398 log_verifier.cc:126] Checking tablet dfba18f2f9fe4d259a369ad071f8c274
I20240503 07:59:59.017657 398 log_verifier.cc:177] Verified matching terms for 723 ops in tablet dfba18f2f9fe4d259a369ad071f8c274
I20240503 07:59:59.017867 398 log_verifier.cc:126] Checking tablet e21fe5b5f5a04168850a773226658450
I20240503 07:59:59.119825 398 log_verifier.cc:177] Verified matching terms for 656 ops in tablet e21fe5b5f5a04168850a773226658450
I20240503 07:59:59.120023 398 log_verifier.cc:126] Checking tablet f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:59.221416 398 log_verifier.cc:177] Verified matching terms for 667 ops in tablet f0088b17e63d431eaa49ef33a4af3b04
I20240503 07:59:59.222847 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6807
I20240503 07:59:59.231078 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6938
I20240503 07:59:59.237648 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 7069
I20240503 07:59:59.243738 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 8050
I20240503 07:59:59.248731 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 8181
I20240503 07:59:59.253605 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 8312
I20240503 07:59:59.258967 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 8443
I20240503 07:59:59.263872 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 11302
I20240503 07:59:59.268640 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 6716
2024-05-03T07:59:59Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (38967 ms)
[----------] 1 test from RebalanceParamTest (38967 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 07:59:59.317657 398 test_util.cc:268] Using random seed: 456062611
I20240503 07:59:59.318445 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:59.318490 398 ts_itest-base.cc:113] --------------
I20240503 07:59:59.318501 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:59:59.318511 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:59.318521 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:59Z Disabled control of system clock
I20240503 07:59:59.326586 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37099
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37099
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 07:59:59.426209 12642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:59.426401 12642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:59.426441 12642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:59.429404 12642 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:59.429461 12642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:59.429486 12642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:59.429523 12642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:59.429551 12642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:59.433195 12642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37099
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:59.433990 12642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:59.435055 12642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:59.442175 12642 server_base.cc:1034] running on GCE node
W20240503 07:59:59.442765 12650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.443295 12647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.443840 12648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:59.444294 12642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:59.445147 12642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:59.446302 12642 hybrid_clock.cc:648] HybridClock initialized: now 1714723199446283 us; error 28 us; skew 500 ppm
I20240503 07:59:59.448401 12642 webserver.cc:460] Webserver started at http://127.0.99.190:35685/ using document root <none> and password file <none>
I20240503 07:59:59.448915 12642 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:59.448966 12642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:59.449127 12642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:59.450690 12642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dc2c479861bf4184ba4d8446b0c8d4cf"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.451184 12642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dc2c479861bf4184ba4d8446b0c8d4cf"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.454284 12642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 07:59:59.456383 12656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.457331 12642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:59.457427 12642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dc2c479861bf4184ba4d8446b0c8d4cf"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.457515 12642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:59.485577 12642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:59.486280 12642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:59.486439 12642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:59.493470 12642 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37099
I20240503 07:59:59.493580 12708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37099 every 8 connection(s)
I20240503 07:59:59.494443 12642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:59.497006 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.501199 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 12642
I20240503 07:59:59.501307 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:59.501809 12709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf: Bootstrap starting.
I20240503 07:59:59.503953 12709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:59.504580 12709 log.cc:824] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:59.505997 12709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf: No bootstrap required, opened a new log
I20240503 07:59:59.508479 12709 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } }
I20240503 07:59:59.508695 12709 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:59.508750 12709 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2c479861bf4184ba4d8446b0c8d4cf, State: Initialized, Role: FOLLOWER
I20240503 07:59:59.509159 12709 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } }
I20240503 07:59:59.509289 12709 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:59.509344 12709 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:59.509445 12709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:59.510105 12709 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } }
I20240503 07:59:59.510423 12709 leader_election.cc:302] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2c479861bf4184ba4d8446b0c8d4cf; no voters:
I20240503 07:59:59.510655 12709 leader_election.cc:288] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:59.510751 12714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:59.510964 12714 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [term 1 LEADER]: Becoming Leader. State: Replica: dc2c479861bf4184ba4d8446b0c8d4cf, State: Running, Role: LEADER
I20240503 07:59:59.511231 12714 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } }
I20240503 07:59:59.511632 12709 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:59.512742 12716 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2c479861bf4184ba4d8446b0c8d4cf. Latest consensus state: current_term: 1 leader_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } } }
I20240503 07:59:59.512776 12715 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2c479861bf4184ba4d8446b0c8d4cf" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37099 } } }
I20240503 07:59:59.512846 12716 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:59.512877 12715 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:59.513615 12720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:59.516057 12720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:59.520995 12720 catalog_manager.cc:1347] Generated new cluster ID: 0b6f0f69663244edb495c1d6769247de
I20240503 07:59:59.521063 12720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:59.531548 12720 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:59.532665 12720 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:59.543932 12720 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dc2c479861bf4184ba4d8446b0c8d4cf: Generated new TSK 0
I20240503 07:59:59.544731 12720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:59.552321 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 07:59:59.655889 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:59.656119 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:59.656181 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:59.659248 12733 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:59.659320 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:59.659402 12733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:59.663100 12733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.12733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:59.663965 12733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:59.665040 12733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:59.673143 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.673751 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.674311 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:59.674993 12733 server_base.cc:1034] running on GCE node
I20240503 07:59:59.675343 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:59.675860 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:59.677003 12733 hybrid_clock.cc:648] HybridClock initialized: now 1714723199676992 us; error 44 us; skew 500 ppm
I20240503 07:59:59.678789 12733 webserver.cc:460] Webserver started at http://127.0.99.129:39843/ using document root <none> and password file <none>
I20240503 07:59:59.679333 12733 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:59.679390 12733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:59.679550 12733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:59.681094 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6c358da9faef48c4adb12cce628aeb38"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.681527 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6c358da9faef48c4adb12cce628aeb38"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.684577 12733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:59.686736 12747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.687747 12733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:59.687851 12733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c358da9faef48c4adb12cce628aeb38"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.687933 12733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:59.701587 12733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:59.702111 12733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:59.702247 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:59.702723 12733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:59.703601 12733 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:59.703645 12733 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.703684 12733 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:59.703711 12733 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.711774 12733 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43847
I20240503 07:59:59.711809 12860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43847 every 8 connection(s)
I20240503 07:59:59.712656 12733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:59.717402 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 12733
I20240503 07:59:59.717527 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:59.719396 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:59.721519 12861 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 07:59:59.721715 12861 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:59.722105 12861 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 07:59:59.723438 12673 ts_manager.cc:194] Registered new tserver with Master: 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 07:59:59.724532 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:53541
W20240503 07:59:59.820511 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:59.820722 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:59.820780 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:59.823729 12864 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 07:59:59.823787 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:59.823869 12864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:59.827351 12864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:59.828161 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:59.829208 12864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:59.837564 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.838182 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:59.838752 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:59.839396 12864 server_base.cc:1034] running on GCE node
I20240503 07:59:59.839795 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:59.840289 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:59.841437 12864 hybrid_clock.cc:648] HybridClock initialized: now 1714723199841421 us; error 30 us; skew 500 ppm
I20240503 07:59:59.843211 12864 webserver.cc:460] Webserver started at http://127.0.99.130:33061/ using document root <none> and password file <none>
I20240503 07:59:59.843710 12864 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:59.843761 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:59.843899 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:59.845404 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0cf63591cfc244028e1fc368c2bca079"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.845798 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0cf63591cfc244028e1fc368c2bca079"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.848841 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:59.850838 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.851779 12864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:59.851879 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0cf63591cfc244028e1fc368c2bca079"
format_stamp: "Formatted at 2024-05-03 07:59:59 on dist-test-slave-11df"
I20240503 07:59:59.851953 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:59.879889 12864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:59.880517 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:59.880668 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:59.881186 12864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:59.882055 12864 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:59.882102 12864 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.882153 12864 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:59.882179 12864 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:59.890200 12864 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33937
I20240503 07:59:59.890229 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33937 every 8 connection(s)
I20240503 07:59:59.891146 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:59.894869 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 12864
I20240503 07:59:59.895015 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:59.897099 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 07:59:59.900158 12992 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 07:59:59.900396 12992 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:59.900803 12992 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 07:59:59.901679 12673 ts_manager.cc:194] Registered new tserver with Master: 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 07:59:59.902184 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55093
W20240503 07:59:59.999063 12995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:59.999297 12995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:59.999361 12995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.002374 12995 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.002450 12995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.002539 12995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:00.006168 12995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.12995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.007076 12995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.008155 12995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.016547 13001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.017107 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.017714 13000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.018316 12995 server_base.cc:1034] running on GCE node
I20240503 08:00:00.018646 12995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.019165 12995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.020315 12995 hybrid_clock.cc:648] HybridClock initialized: now 1714723200020295 us; error 32 us; skew 500 ppm
I20240503 08:00:00.022111 12995 webserver.cc:460] Webserver started at http://127.0.99.131:36541/ using document root <none> and password file <none>
I20240503 08:00:00.022634 12995 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.022686 12995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.022847 12995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.024478 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "39827620daed48cb8f137442e1f1da76"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.024900 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "39827620daed48cb8f137442e1f1da76"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.028061 12995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:00.030197 13009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.031270 12995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:00.031373 12995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "39827620daed48cb8f137442e1f1da76"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.031455 12995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.049633 12995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.050287 12995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.050455 12995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.051035 12995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.051949 12995 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.051996 12995 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.052048 12995 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.052073 12995 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.059943 12995 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43153
I20240503 08:00:00.059967 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43153 every 8 connection(s)
I20240503 08:00:00.060870 12995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:00.062656 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 12995
I20240503 08:00:00.062775 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:00.064785 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:00.069531 13123 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.069744 13123 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.070159 13123 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.071135 12673 ts_manager.cc:194] Registered new tserver with Master: 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:00.071847 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:53315
W20240503 08:00:00.167613 13126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:00.167816 13126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:00.167857 13126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.170810 13126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.170873 13126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.170977 13126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:00.174465 13126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.13126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.175284 13126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.176307 13126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.184291 13134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.184834 13131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.185415 13132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.186025 13126 server_base.cc:1034] running on GCE node
I20240503 08:00:00.186383 13126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.186939 13126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.188095 13126 hybrid_clock.cc:648] HybridClock initialized: now 1714723200188078 us; error 30 us; skew 500 ppm
I20240503 08:00:00.189894 13126 webserver.cc:460] Webserver started at http://127.0.99.132:35557/ using document root <none> and password file <none>
I20240503 08:00:00.190444 13126 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.190498 13126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.190655 13126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.192323 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "af712db2f85249adbe718e8ba0549f84"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.192757 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "af712db2f85249adbe718e8ba0549f84"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.195885 13126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 08:00:00.197962 13140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.198969 13126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:00.199074 13126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "af712db2f85249adbe718e8ba0549f84"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.199162 13126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.226083 13126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.226737 13126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.226897 13126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.227494 13126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.228392 13126 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.228448 13126 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.228495 13126 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.228523 13126 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.237107 13126 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37969
I20240503 08:00:00.237129 13253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37969 every 8 connection(s)
I20240503 08:00:00.238008 13126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:00.240087 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 13126
I20240503 08:00:00.240216 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:00.242221 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:00.247164 13254 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.247378 13254 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.247834 13254 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.248818 12673 ts_manager.cc:194] Registered new tserver with Master: af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:00.249543 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:38931
W20240503 08:00:00.347255 13257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:00.347465 13257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:00.347508 13257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.350471 13257 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.350525 13257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.350589 13257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:00:00.354122 13257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.13257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.355023 13257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.356201 13257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.364547 13262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.365139 13263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.365677 13265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.366269 13257 server_base.cc:1034] running on GCE node
I20240503 08:00:00.366664 13257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.367287 13257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.368446 13257 hybrid_clock.cc:648] HybridClock initialized: now 1714723200368439 us; error 34 us; skew 500 ppm
I20240503 08:00:00.370287 13257 webserver.cc:460] Webserver started at http://127.0.99.133:41555/ using document root <none> and password file <none>
I20240503 08:00:00.370842 13257 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.370899 13257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.371088 13257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.372714 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "82cd33f2bae4419b946095f8eb93e1e4"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.373157 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "82cd33f2bae4419b946095f8eb93e1e4"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.376454 13257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:00:00.378592 13271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.379633 13257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:00.379734 13257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "82cd33f2bae4419b946095f8eb93e1e4"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.379812 13257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.402889 13257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.403556 13257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.403717 13257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.404227 13257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.405117 13257 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.405164 13257 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.405215 13257 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.405243 13257 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.413139 13257 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38311
I20240503 08:00:00.413157 13384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38311 every 8 connection(s)
I20240503 08:00:00.414105 13257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:00:00.418100 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 13257
I20240503 08:00:00.418203 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:00:00.420147 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:00.422353 13385 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.422564 13385 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.423036 13385 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.423914 12673 ts_manager.cc:194] Registered new tserver with Master: 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:00.424417 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:49399
W20240503 08:00:00.531942 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:00.532164 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:00.532208 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.535358 13388 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.535432 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.535526 13388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 08:00:00.539162 13388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.13388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.540027 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.541136 13388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.549639 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.550248 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.550793 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.551463 13388 server_base.cc:1034] running on GCE node
I20240503 08:00:00.551880 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.552443 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.553607 13388 hybrid_clock.cc:648] HybridClock initialized: now 1714723200553582 us; error 39 us; skew 500 ppm
I20240503 08:00:00.555650 13388 webserver.cc:460] Webserver started at http://127.0.99.134:38591/ using document root <none> and password file <none>
I20240503 08:00:00.556226 13388 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.556284 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.556455 13388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.558064 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8398e6f89bf5419d81ed5bed991da0cf"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.558519 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8398e6f89bf5419d81ed5bed991da0cf"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.561928 13388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240503 08:00:00.564194 13402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.565151 13388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:00.565255 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8398e6f89bf5419d81ed5bed991da0cf"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.565337 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.584206 13388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.584825 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.584980 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.585520 13388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.586402 13388 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.586457 13388 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.586509 13388 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.586531 13388 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.594676 13388 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39483
I20240503 08:00:00.594699 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39483 every 8 connection(s)
I20240503 08:00:00.595603 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:00:00.596066 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 13388
I20240503 08:00:00.596190 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:00:00.599131 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:00.604888 13516 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.605086 13516 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.605499 13516 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.606375 12673 ts_manager.cc:194] Registered new tserver with Master: 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:00.607126 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:44897
W20240503 08:00:00.705098 13519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:00.705313 13519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:00.705356 13519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.708313 13519 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.708376 13519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.708447 13519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 08:00:00.712173 13519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.13519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.713181 13519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.714720 13519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.723124 13525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.723698 13524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.724247 13527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.724870 13519 server_base.cc:1034] running on GCE node
I20240503 08:00:00.725234 13519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.725771 13519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.726462 12861 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:00.726955 13519 hybrid_clock.cc:648] HybridClock initialized: now 1714723200726929 us; error 38 us; skew 500 ppm
I20240503 08:00:00.729056 13519 webserver.cc:460] Webserver started at http://127.0.99.135:33423/ using document root <none> and password file <none>
I20240503 08:00:00.729604 13519 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.729665 13519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.729830 13519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.731587 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b052e89b63c24258872c832a6a0d4d4d"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.732056 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b052e89b63c24258872c832a6a0d4d4d"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.735530 13519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:00.737916 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.738898 13519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:00.739050 13519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b052e89b63c24258872c832a6a0d4d4d"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.739140 13519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.753381 13519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.753939 13519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.754078 13519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.754575 13519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.755494 13519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.755539 13519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.755580 13519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.755606 13519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.763653 13519 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37289
I20240503 08:00:00.763684 13646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37289 every 8 connection(s)
I20240503 08:00:00.764537 13519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:00:00.765631 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 13519
I20240503 08:00:00.765743 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:00:00.767743 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:00.772948 13647 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.773138 13647 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.773557 13647 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.774396 12673 ts_manager.cc:194] Registered new tserver with Master: b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:00.774904 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:38441
W20240503 08:00:00.872759 13650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:00.872958 13650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:00.873008 13650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:00.875991 13650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:00.876048 13650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:00.876123 13650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:00.879738 13650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.13650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:00.880618 13650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:00.881731 13650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:00.890391 13656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.890957 13655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:00.891554 13658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:00.892251 13650 server_base.cc:1034] running on GCE node
I20240503 08:00:00.892701 13650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:00.893352 13650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:00.894521 13650 hybrid_clock.cc:648] HybridClock initialized: now 1714723200894496 us; error 54 us; skew 500 ppm
I20240503 08:00:00.896330 13650 webserver.cc:460] Webserver started at http://127.0.99.136:39099/ using document root <none> and password file <none>
I20240503 08:00:00.896849 13650 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:00.896898 13650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:00.897037 13650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:00.898577 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "eae9f53215a34bc88abe7b3c1c223319"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.899027 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "eae9f53215a34bc88abe7b3c1c223319"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.901988 13650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:00.903988 12992 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:00.904151 13664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.905247 13650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:00.905346 13650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "eae9f53215a34bc88abe7b3c1c223319"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:00.905417 13650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:00.934294 13650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:00.934899 13650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:00.935103 13650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:00.935618 13650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:00.936504 13650 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:00.936547 13650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.936585 13650 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:00.936604 13650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:00.944458 13650 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35323
I20240503 08:00:00.944504 13777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35323 every 8 connection(s)
I20240503 08:00:00.945353 13650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:00.953888 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 13650
I20240503 08:00:00.954006 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:00:00.954141 13778 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:00.954360 13778 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:00.954772 13778 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:00.955682 12673 ts_manager.cc:194] Registered new tserver with Master: eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:00.956260 12673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:59169
I20240503 08:00:00.958184 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:00:00.971774 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:00:00.971873 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 13257
I20240503 08:00:00.976681 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 13388
I20240503 08:00:00.981734 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 13519
I20240503 08:00:00.985946 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 13650
I20240503 08:00:01.073738 13123 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:01.251603 13254 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:04.741595 12673 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:04.761444 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7f229898c24572815177628720521f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.761444 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 20cc823a5ce34916894cfde0a6614bd1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.761741 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 20cc823a5ce34916894cfde0a6614bd1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.762403 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.762403 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cc823a5ce34916894cfde0a6614bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.762568 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cc823a5ce34916894cfde0a6614bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.763473 12794 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7bf40aa0614923a19feed8bab430fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.763228 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7bf40aa0614923a19feed8bab430fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.763689 12794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7bf40aa0614923a19feed8bab430fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.764014 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7bf40aa0614923a19feed8bab430fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.761445 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7bf40aa0614923a19feed8bab430fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.767270 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7bf40aa0614923a19feed8bab430fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.763222 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 20cc823a5ce34916894cfde0a6614bd1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.767482 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cc823a5ce34916894cfde0a6614bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.767760 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7f229898c24572815177628720521f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.767989 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.769004 12925 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7f229898c24572815177628720521f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:04.769228 12925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:04.771358 13800 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:04.771399 13799 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:04.772696 13801 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:04.773443 13800 tablet_bootstrap.cc:654] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.774174 13799 tablet_bootstrap.cc:654] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.774250 13800 log.cc:824] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:04.774554 13801 tablet_bootstrap.cc:654] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.774937 13799 log.cc:824] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:04.775077 13798 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:04.775368 13801 log.cc:824] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:04.775940 13800 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:04.776108 13800 ts_tablet_manager.cc:1367] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:00:04.776700 13799 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:04.776808 13801 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:04.776840 13799 ts_tablet_manager.cc:1367] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:00:04.776897 13798 tablet_bootstrap.cc:654] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.776952 13801 ts_tablet_manager.cc:1367] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:04.777568 13798 log.cc:824] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:04.779104 13798 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:04.779250 13798 ts_tablet_manager.cc:1367] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:00:04.779731 13800 raft_consensus.cc:348] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.779930 13800 raft_consensus.cc:374] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.779975 13800 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.780403 13800 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.780398 13801 raft_consensus.cc:348] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.780575 13801 raft_consensus.cc:374] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.780615 13801 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.781020 13801 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.781255 13800 ts_tablet_manager.cc:1398] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:04.781335 13799 raft_consensus.cc:348] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.781534 13799 raft_consensus.cc:374] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.781580 13799 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.781610 13800 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:04.781850 13801 ts_tablet_manager.cc:1398] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 08:00:04.781931 13799 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.782163 13801 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:04.782671 13799 ts_tablet_manager.cc:1398] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:00:04.783203 13801 tablet_bootstrap.cc:654] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.783514 13800 tablet_bootstrap.cc:654] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.784404 13799 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:04.784740 13801 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:04.785135 13801 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:04.786011 13798 raft_consensus.cc:348] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.786243 13798 raft_consensus.cc:374] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.786322 13798 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.785878 13799 tablet_bootstrap.cc:654] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.786648 13800 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:04.786732 13800 ts_tablet_manager.cc:1367] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:04.786729 13798 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.787258 13800 raft_consensus.cc:348] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.787353 13800 raft_consensus.cc:374] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.787384 13800 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.787533 13798 ts_tablet_manager.cc:1398] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:00:04.787511 13800 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.787637 13801 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.787724 13801 raft_consensus.cc:374] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.787753 13801 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.787763 13800 ts_tablet_manager.cc:1398] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:04.787842 13798 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:04.787863 13801 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.788117 13801 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 08:00:04.788691 13798 tablet_bootstrap.cc:654] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.789939 13799 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:04.790033 13799 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:04.790288 13798 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:04.790431 13798 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:04.790557 13799 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.790670 13799 raft_consensus.cc:374] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.790710 13799 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.790823 13799 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.791059 13798 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.791108 13799 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:04.791157 13798 raft_consensus.cc:374] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.791185 13798 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.791287 13799 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:04.791390 13798 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.791712 13798 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:04.792064 13799 tablet_bootstrap.cc:654] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:04.793098 13799 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:04.793169 13799 ts_tablet_manager.cc:1367] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:04.793569 13799 raft_consensus.cc:348] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.793665 13799 raft_consensus.cc:374] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:04.793706 13799 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:04.793809 13799 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.794035 13799 ts_tablet_manager.cc:1398] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 08:00:04.814777 13124 tablet.cc:2369] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:04.828227 13806 raft_consensus.cc:482] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:04.828425 13806 raft_consensus.cc:504] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.829542 13806 leader_election.cc:288] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:04.834179 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:04.834177 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:04.834465 12946 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:04.834465 12815 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:04.834882 13144 leader_election.cc:302] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:04.835162 13806 raft_consensus.cc:2781] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:04.835234 13806 raft_consensus.cc:482] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:04.835290 13806 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.836269 13806 raft_consensus.cc:504] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.836601 13806 leader_election.cc:288] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:04.836918 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:04.836918 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:04.837038 12815 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.837059 12946 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.838112 12815 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:04.838392 12946 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:04.838747 13143 leader_election.cc:302] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:04.838897 13806 raft_consensus.cc:2781] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:04.839100 13806 raft_consensus.cc:686] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:04.839385 13806 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:04.842043 12670 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:04.845360 13806 raft_consensus.cc:482] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:04.845440 13806 raft_consensus.cc:504] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.845746 13806 leader_election.cc:288] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:04.846064 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:04.846189 12946 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:04.846428 13144 leader_election.cc:302] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:04.846578 13806 raft_consensus.cc:2781] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:04.846619 13806 raft_consensus.cc:482] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:04.846663 13806 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.847400 13806 raft_consensus.cc:504] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.847687 13806 leader_election.cc:288] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:04.847949 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:04.848073 12946 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.848788 12946 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:04.849007 13144 leader_election.cc:302] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:04.849205 13806 raft_consensus.cc:2781] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:04.849153 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:04.849154 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:04.849363 13806 raft_consensus.cc:686] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:04.849431 13077 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:04.849486 13806 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:04.850955 12670 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:04.863333 13808 raft_consensus.cc:482] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:04.863431 13808 raft_consensus.cc:504] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.864356 13808 leader_election.cc:288] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:04.867796 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:04.867849 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:04.867937 13076 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:04.867952 12815 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:04.868242 12879 leader_election.cc:302] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:04.868453 13808 raft_consensus.cc:2781] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:04.868510 13808 raft_consensus.cc:482] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:04.868552 13808 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.869261 13808 raft_consensus.cc:504] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.869549 13808 leader_election.cc:288] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:04.869848 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:04.869880 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:04.869962 13076 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.869974 12815 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:04.870636 12815 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 1.
I20240503 08:00:04.870780 13076 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 1.
I20240503 08:00:04.870884 12881 leader_election.cc:302] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:04.871063 13808 raft_consensus.cc:2781] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:04.871259 13808 raft_consensus.cc:686] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:04.871539 13808 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:04.874032 12670 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
W20240503 08:00:04.895290 12993 tablet.cc:2369] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:04.967006 12862 tablet.cc:2369] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:04.981922 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:04.982137 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:04.985086 13824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 08:00:04.992161 13255 tablet.cc:2369] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
Using table 'rebalance_test_table_0'
I20240503 08:00:05.063112 12815 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.063122 13076 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.063594 13077 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.063974 13808 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.063993 12946 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.064647 13077 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.064937 13806 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.065083 13817 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.066136 13808 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.070068 12945 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.070442 13806 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.072376 12813 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.072712 13806 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.078975 13854 mvcc.cc:205] Tried to move back new op lower bound from 7023506247933210624 to 7023506247024508928. Current Snapshot: MvccSnapshot[applied={T|T < 7023506247933210624}]
I20240503 08:00:05.082762 13853 mvcc.cc:205] Tried to move back new op lower bound from 7023506247934455808 to 7023506247064883200. Current Snapshot: MvccSnapshot[applied={T|T < 7023506247934455808}]
I20240503 08:00:05.085152 13857 mvcc.cc:205] Tried to move back new op lower bound from 7023506247928188928 to 7023506247155752960. Current Snapshot: MvccSnapshot[applied={T|T < 7023506247928188928}]
I20240503 08:00:05.090312 13860 mvcc.cc:205] Tried to move back new op lower bound from 7023506247928188928 to 7023506247155752960. Current Snapshot: MvccSnapshot[applied={T|T < 7023506247928188928}]
INSERT report
rows total: 2000
time total: 105.271 ms
time per row: 0.0526356 ms
I20240503 08:00:05.131760 12672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:05.137030 13056 tablet_service.cc:1467] Processing CreateTablet for tablet a924487225dc4246aeb695060084096b (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.137259 12794 tablet_service.cc:1467] Processing CreateTablet for tablet 672fc9981d074f7e8f2324ba0b06be57 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.137385 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a924487225dc4246aeb695060084096b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.137552 12794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672fc9981d074f7e8f2324ba0b06be57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.137605 12795 tablet_service.cc:1467] Processing CreateTablet for tablet a924487225dc4246aeb695060084096b (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.137908 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a924487225dc4246aeb695060084096b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.138584 13188 tablet_service.cc:1467] Processing CreateTablet for tablet a924487225dc4246aeb695060084096b (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.138830 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a924487225dc4246aeb695060084096b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.139634 13798 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:05.139768 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 672fc9981d074f7e8f2324ba0b06be57 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.139966 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672fc9981d074f7e8f2324ba0b06be57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.140348 13801 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:05.140794 13800 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:05.141275 12926 tablet_service.cc:1467] Processing CreateTablet for tablet d8994c38d0f640e9bd5b5c322c603b9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.141530 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.141660 13800 tablet_bootstrap.cc:654] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.142146 12795 tablet_service.cc:1467] Processing CreateTablet for tablet d8994c38d0f640e9bd5b5c322c603b9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.142359 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.142673 13800 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:05.142755 13800 ts_tablet_manager.cc:1367] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.143218 13800 raft_consensus.cc:348] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.143226 13057 tablet_service.cc:1467] Processing CreateTablet for tablet d8994c38d0f640e9bd5b5c322c603b9f (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.143319 13800 raft_consensus.cc:374] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.143350 13800 raft_consensus.cc:729] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.143451 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.143484 13799 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:05.143458 13800 consensus_queue.cc:260] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.143700 13800 ts_tablet_manager.cc:1398] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.143851 13800 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:05.144234 13799 tablet_bootstrap.cc:654] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.144606 13800 tablet_bootstrap.cc:654] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.144731 13798 tablet_bootstrap.cc:654] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.145349 13799 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:05.145419 13799 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.145571 13801 tablet_bootstrap.cc:654] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.145835 13799 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.145931 13799 raft_consensus.cc:374] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.145959 13799 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.146067 13799 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.146275 13799 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:05.146340 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 672fc9981d074f7e8f2324ba0b06be57 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.146546 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672fc9981d074f7e8f2324ba0b06be57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.146790 13800 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:05.146845 13800 ts_tablet_manager.cc:1367] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:05.147251 13800 raft_consensus.cc:348] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.147356 13800 raft_consensus.cc:374] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.147392 13800 raft_consensus.cc:729] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.147522 13800 consensus_queue.cc:260] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.147598 13801 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:05.147673 13801 ts_tablet_manager.cc:1367] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:05.147768 13800 ts_tablet_manager.cc:1398] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:05.148085 13801 raft_consensus.cc:348] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.148193 13801 raft_consensus.cc:374] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.148226 13801 raft_consensus.cc:729] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.148350 13801 consensus_queue.cc:260] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.148604 13801 ts_tablet_manager.cc:1398] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:05.148761 13801 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:05.149538 13801 tablet_bootstrap.cc:654] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.151091 13798 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:05.151108 13801 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:05.151166 13801 ts_tablet_manager.cc:1367] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.151175 13798 ts_tablet_manager.cc:1367] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:00:05.151522 13801 raft_consensus.cc:348] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.151605 13801 raft_consensus.cc:374] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.151634 13801 raft_consensus.cc:729] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.151589 13798 raft_consensus.cc:348] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.151681 13798 raft_consensus.cc:374] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.151713 13798 raft_consensus.cc:729] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.151840 13798 consensus_queue.cc:260] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.151880 13801 consensus_queue.cc:260] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.152081 13798 ts_tablet_manager.cc:1398] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.152251 13798 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:05.152333 13801 ts_tablet_manager.cc:1398] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.152475 13801 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:05.153096 13798 tablet_bootstrap.cc:654] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.153303 13801 tablet_bootstrap.cc:654] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.154314 13801 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:05.154386 13801 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.155007 13798 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:05.155009 13801 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.155086 13801 raft_consensus.cc:374] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.155097 13798 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:05.155121 13801 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.155234 13801 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.155472 13798 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.155506 13801 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.155555 13798 raft_consensus.cc:374] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.155591 13798 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.155799 13798 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.156430 13798 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:05.157004 13798 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:05.159921 13798 tablet_bootstrap.cc:654] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.161005 13798 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:05.161088 13798 ts_tablet_manager.cc:1367] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:00:05.161386 13798 raft_consensus.cc:348] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.161463 13798 raft_consensus.cc:374] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.161489 13798 raft_consensus.cc:729] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.161574 13798 consensus_queue.cc:260] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.161759 13798 ts_tablet_manager.cc:1398] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:05.203933 13812 raft_consensus.cc:482] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.204099 13812 raft_consensus.cc:504] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.205157 13812 leader_election.cc:288] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:05.209010 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672fc9981d074f7e8f2324ba0b06be57" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:05.209091 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672fc9981d074f7e8f2324ba0b06be57" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:05.209187 12813 raft_consensus.cc:2454] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:05.209286 13208 raft_consensus.cc:2454] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:05.209545 13012 leader_election.cc:302] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.209808 13812 raft_consensus.cc:2781] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.209869 13812 raft_consensus.cc:482] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.209894 13812 raft_consensus.cc:3037] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.210691 13812 raft_consensus.cc:504] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.211045 13812 leader_election.cc:288] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:05.211346 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672fc9981d074f7e8f2324ba0b06be57" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:05.211372 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672fc9981d074f7e8f2324ba0b06be57" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:05.211472 13208 raft_consensus.cc:3037] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.211472 12813 raft_consensus.cc:3037] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.212430 13208 raft_consensus.cc:2454] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:05.212430 12813 raft_consensus.cc:2454] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:05.212734 13012 leader_election.cc:302] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.212913 13812 raft_consensus.cc:2781] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.213111 13812 raft_consensus.cc:686] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:05.213449 13812 consensus_queue.cc:237] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.215569 12672 catalog_manager.cc:5484] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:05.235121 13807 raft_consensus.cc:482] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.235278 13807 raft_consensus.cc:504] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.236351 13807 leader_election.cc:288] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:05.240329 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:05.240487 13077 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.240547 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:05.240672 12945 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.240820 12748 leader_election.cc:302] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.241036 13807 raft_consensus.cc:2781] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.241096 13807 raft_consensus.cc:482] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.241132 13807 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.241981 13807 raft_consensus.cc:504] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.242303 13807 leader_election.cc:288] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:05.242569 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:05.242666 12945 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.242794 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:05.242885 13077 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.243675 13077 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:05.243747 12945 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:05.243916 12748 leader_election.cc:302] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.244060 13807 raft_consensus.cc:2781] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.244261 13807 raft_consensus.cc:686] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:05.244546 13807 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.246373 12672 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:00:05.493252 13807 raft_consensus.cc:482] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.493409 13807 raft_consensus.cc:504] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.493995 13807 leader_election.cc:288] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:05.494542 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a924487225dc4246aeb695060084096b" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:05.494701 13077 raft_consensus.cc:2454] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.495170 12748 leader_election.cc:302] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.495353 13807 raft_consensus.cc:2781] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.495405 13807 raft_consensus.cc:482] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.495442 13807 raft_consensus.cc:3037] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.496488 13807 raft_consensus.cc:504] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.497072 13807 leader_election.cc:288] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:05.497506 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a924487225dc4246aeb695060084096b" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:05.497633 13077 raft_consensus.cc:3037] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.497885 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a924487225dc4246aeb695060084096b" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:05.497903 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a924487225dc4246aeb695060084096b" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:05.498059 13208 raft_consensus.cc:2454] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.498620 13077 raft_consensus.cc:2454] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:05.498863 12748 leader_election.cc:302] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters: af712db2f85249adbe718e8ba0549f84
I20240503 08:00:05.499064 13807 raft_consensus.cc:2781] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.499151 13807 raft_consensus.cc:686] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:05.499289 13807 consensus_queue.cc:237] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:05.500727 12672 catalog_manager.cc:5484] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
W20240503 08:00:05.608675 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:05.608891 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:05.611845 13903 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:00:05.684334 12813 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.684455 13207 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.684798 13812 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.685153 13812 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.686851 13208 raft_consensus.cc:3037] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.687085 13077 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.687913 13208 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.688988 13807 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.689177 13900 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.694074 13077 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.694089 12945 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:05.694478 13900 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:05.694882 13900 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 85.3798 ms
time per row: 0.0426899 ms
I20240503 08:00:05.741480 12672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:05.747558 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.747749 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.747854 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.747972 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.748610 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 70c5de226c7a4b349eef86a79d172851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.748817 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.750108 13187 tablet_service.cc:1467] Processing CreateTablet for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.750120 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.750325 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.750324 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376ac1ea05c482ca8f03ddf0f5fcd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.751348 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 70c5de226c7a4b349eef86a79d172851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.751585 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.752543 13959 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:05.753191 13959 tablet_bootstrap.cc:654] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.753502 13960 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:05.753867 12795 tablet_service.cc:1467] Processing CreateTablet for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.754092 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376ac1ea05c482ca8f03ddf0f5fcd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.754324 12924 tablet_service.cc:1467] Processing CreateTablet for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.754562 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376ac1ea05c482ca8f03ddf0f5fcd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.756104 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 70c5de226c7a4b349eef86a79d172851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:05.756319 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.756888 13960 tablet_bootstrap.cc:654] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.757957 13960 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:05.758051 13960 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:05.758270 13957 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:05.758520 13960 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.758641 13960 raft_consensus.cc:374] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.758680 13960 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.758806 13960 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.759115 13960 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.759217 13959 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:05.759290 13959 ts_tablet_manager.cc:1367] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:05.759737 13959 raft_consensus.cc:348] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.759837 13959 raft_consensus.cc:374] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.759868 13959 raft_consensus.cc:729] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.759979 13959 consensus_queue.cc:260] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.760216 13959 ts_tablet_manager.cc:1398] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.760341 13959 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:05.760673 13957 tablet_bootstrap.cc:654] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.761133 13959 tablet_bootstrap.cc:654] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.762329 13959 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:05.762420 13959 ts_tablet_manager.cc:1367] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:05.762837 13959 raft_consensus.cc:348] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.762957 13959 raft_consensus.cc:374] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.762990 13959 raft_consensus.cc:729] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.763103 13959 consensus_queue.cc:260] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.762441 13958 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:05.763353 13959 ts_tablet_manager.cc:1398] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.763984 13957 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:05.764068 13957 ts_tablet_manager.cc:1367] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:05.764421 13958 tablet_bootstrap.cc:654] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.764415 13957 raft_consensus.cc:348] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.764499 13957 raft_consensus.cc:374] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.764536 13957 raft_consensus.cc:729] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.764683 13957 consensus_queue.cc:260] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.765025 13957 ts_tablet_manager.cc:1398] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.765204 13957 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:05.765527 13958 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:05.765621 13958 ts_tablet_manager.cc:1367] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:05.766049 13957 tablet_bootstrap.cc:654] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.766067 13958 raft_consensus.cc:348] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.766170 13958 raft_consensus.cc:374] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.766209 13958 raft_consensus.cc:729] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.766345 13958 consensus_queue.cc:260] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.766598 13958 ts_tablet_manager.cc:1398] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.766754 13958 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:05.767127 13957 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:05.767628 13958 tablet_bootstrap.cc:654] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.767664 13957 ts_tablet_manager.cc:1367] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:05.768040 13957 raft_consensus.cc:348] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.768131 13957 raft_consensus.cc:374] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.768159 13957 raft_consensus.cc:729] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.768267 13957 consensus_queue.cc:260] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.768498 13957 ts_tablet_manager.cc:1398] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:05.768648 13957 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:05.768784 13958 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:05.768857 13958 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.769264 13958 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.769423 13957 tablet_bootstrap.cc:654] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.769472 13958 raft_consensus.cc:374] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.769528 13958 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.769654 13958 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.770177 13958 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.770362 13958 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:05.770565 13957 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:05.770648 13957 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:05.771073 13957 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.771257 13957 raft_consensus.cc:374] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.771301 13957 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.771319 13958 tablet_bootstrap.cc:654] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:05.771418 13957 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.771733 13957 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:05.772428 13958 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:05.772529 13958 ts_tablet_manager.cc:1367] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:05.773003 13958 raft_consensus.cc:348] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.773125 13958 raft_consensus.cc:374] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:05.773169 13958 raft_consensus.cc:729] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:05.773303 13958 consensus_queue.cc:260] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.773581 13958 ts_tablet_manager.cc:1398] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:05.779914 13806 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.780004 13806 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.780328 13806 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:05.780654 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:05.780779 12945 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:05.780757 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:05.780879 13076 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:05.781042 13144 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:05.781198 13806 raft_consensus.cc:2781] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.781255 13806 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.781288 13806 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.782053 13806 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.782349 13806 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:05.782630 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:05.782655 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:05.782723 12945 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.782783 13076 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.783499 12945 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:05.783524 13076 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:05.783759 13144 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:05.783916 13806 raft_consensus.cc:2781] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.784111 13806 raft_consensus.cc:686] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:05.784238 13806 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:05.785688 12670 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:00:05.807446 13900 raft_consensus.cc:482] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.807577 13900 raft_consensus.cc:504] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.807971 13900 leader_election.cc:288] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:05.808418 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3fc1d0e6a42f38406bb75f69b8cbc" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:05.808406 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3fc1d0e6a42f38406bb75f69b8cbc" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:05.808549 13208 raft_consensus.cc:2454] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.808549 12945 raft_consensus.cc:2454] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:05.808871 12751 leader_election.cc:302] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.809048 13900 raft_consensus.cc:2781] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.809108 13900 raft_consensus.cc:482] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.809131 13900 raft_consensus.cc:3037] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.809947 13900 raft_consensus.cc:504] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.810252 13900 leader_election.cc:288] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:05.810559 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3fc1d0e6a42f38406bb75f69b8cbc" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:05.810561 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3fc1d0e6a42f38406bb75f69b8cbc" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:05.810655 13208 raft_consensus.cc:3037] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.810670 12945 raft_consensus.cc:3037] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.811406 12945 raft_consensus.cc:2454] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:05.811409 13208 raft_consensus.cc:2454] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:05.811650 12751 leader_election.cc:302] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:05.811883 13900 raft_consensus.cc:2781] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.811954 13900 raft_consensus.cc:686] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:05.812083 13900 consensus_queue.cc:237] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.813674 12670 catalog_manager.cc:5484] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:05.832998 13806 raft_consensus.cc:482] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:05.833115 13806 raft_consensus.cc:504] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.833472 13806 leader_election.cc:288] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:05.833835 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376ac1ea05c482ca8f03ddf0f5fcd47" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:05.833834 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376ac1ea05c482ca8f03ddf0f5fcd47" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:05.833956 12945 raft_consensus.cc:2454] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:05.833973 12813 raft_consensus.cc:2454] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:05.834216 13144 leader_election.cc:302] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:05.834461 13806 raft_consensus.cc:2781] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:05.834520 13806 raft_consensus.cc:482] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:05.834558 13806 raft_consensus.cc:3037] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.835532 13806 raft_consensus.cc:504] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.835829 13806 leader_election.cc:288] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:05.836129 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376ac1ea05c482ca8f03ddf0f5fcd47" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:05.836143 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376ac1ea05c482ca8f03ddf0f5fcd47" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:05.836226 12945 raft_consensus.cc:3037] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.836257 12813 raft_consensus.cc:3037] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:05.836922 12945 raft_consensus.cc:2454] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:05.836952 12813 raft_consensus.cc:2454] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:05.837186 13144 leader_election.cc:302] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:05.837348 13806 raft_consensus.cc:2781] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:05.837425 13806 raft_consensus.cc:686] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:05.837556 13806 consensus_queue.cc:237] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:05.839123 12670 catalog_manager.cc:5484] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
W20240503 08:00:05.970949 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:05.971283 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:05.974745 13974 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:00:06.053406 12945 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.053436 12813 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.053833 13970 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.054198 13970 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.058323 13208 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.058720 13943 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.058857 12946 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.059396 13943 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.060631 13076 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.060645 12944 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.060997 13806 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.061260 13806 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 93.7919 ms
time per row: 0.046896 ms
I20240503 08:00:06.116897 12665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:06.122277 13056 tablet_service.cc:1467] Processing CreateTablet for tablet b094d04434d24923806b0e9e209c3fca (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.122432 12926 tablet_service.cc:1467] Processing CreateTablet for tablet b094d04434d24923806b0e9e209c3fca (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.122614 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b094d04434d24923806b0e9e209c3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.122689 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b094d04434d24923806b0e9e209c3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.123050 12795 tablet_service.cc:1467] Processing CreateTablet for tablet b094d04434d24923806b0e9e209c3fca (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.123276 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b094d04434d24923806b0e9e209c3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.124284 13960 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:06.124491 12794 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3e4066cd14aa18619310a4490246b (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.124689 12794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3e4066cd14aa18619310a4490246b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.124940 12926 tablet_service.cc:1467] Processing CreateTablet for tablet c917c5849f54411eb55d69ab33456bf4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.125094 13960 tablet_bootstrap.cc:654] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.125165 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.125268 13959 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:06.125584 13957 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:06.126019 13959 tablet_bootstrap.cc:654] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.126344 13957 tablet_bootstrap.cc:654] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.127331 13057 tablet_service.cc:1467] Processing CreateTablet for tablet c917c5849f54411eb55d69ab33456bf4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.127475 13957 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:06.127434 13188 tablet_service.cc:1467] Processing CreateTablet for tablet c917c5849f54411eb55d69ab33456bf4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.127547 13957 ts_tablet_manager.cc:1367] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:06.127562 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.127667 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.127439 13187 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3e4066cd14aa18619310a4490246b (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.127836 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3e4066cd14aa18619310a4490246b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.127923 13957 raft_consensus.cc:348] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.128039 13957 raft_consensus.cc:374] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.128072 13957 raft_consensus.cc:729] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.128197 13957 consensus_queue.cc:260] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.128451 13957 ts_tablet_manager.cc:1398] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.128602 13957 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:06.129338 13957 tablet_bootstrap.cc:654] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.129693 13958 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:06.130456 13958 tablet_bootstrap.cc:654] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.130553 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3e4066cd14aa18619310a4490246b (DEFAULT_TABLE table=rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.130762 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3e4066cd14aa18619310a4490246b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.130767 13957 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:06.130828 13957 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:06.131258 13957 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.131366 13957 raft_consensus.cc:374] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.131398 13957 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.131531 13957 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.131862 13957 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:06.132040 13958 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:06.132136 13958 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:06.132794 13959 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:06.132894 13959 ts_tablet_manager.cc:1367] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:06.133312 13959 raft_consensus.cc:348] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.133432 13959 raft_consensus.cc:374] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.133467 13959 raft_consensus.cc:729] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.133529 13960 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:06.133608 13960 ts_tablet_manager.cc:1367] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:00:06.133579 13959 consensus_queue.cc:260] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.133829 13959 ts_tablet_manager.cc:1398] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.133957 13960 raft_consensus.cc:348] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.134006 13959 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:06.134045 13960 raft_consensus.cc:374] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.134079 13960 raft_consensus.cc:729] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.134204 13960 consensus_queue.cc:260] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.134464 13960 ts_tablet_manager.cc:1398] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.134598 13960 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:06.134801 13959 tablet_bootstrap.cc:654] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.135360 13960 tablet_bootstrap.cc:654] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.135618 13958 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.135721 13958 raft_consensus.cc:374] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.135751 13958 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.135851 13958 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.136106 13958 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:06.136267 13958 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:06.137095 13958 tablet_bootstrap.cc:654] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.139427 13958 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:06.139501 13958 ts_tablet_manager.cc:1367] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:06.139838 13958 raft_consensus.cc:348] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.139923 13958 raft_consensus.cc:374] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.139951 13958 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.140072 13958 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.140314 13958 ts_tablet_manager.cc:1398] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.140885 13959 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:06.140978 13959 ts_tablet_manager.cc:1367] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:06.141086 13960 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:06.141175 13960 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:06.141402 13959 raft_consensus.cc:348] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.141518 13959 raft_consensus.cc:374] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.141557 13959 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.141634 13960 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.141672 13959 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.141738 13960 raft_consensus.cc:374] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.141762 13960 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.141871 13960 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.141927 13959 ts_tablet_manager.cc:1398] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.142133 13960 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:06.142298 13960 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:06.143180 13960 tablet_bootstrap.cc:654] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.144794 13960 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:06.144887 13960 ts_tablet_manager.cc:1367] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:06.145370 13960 raft_consensus.cc:348] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.145464 13960 raft_consensus.cc:374] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.145495 13960 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.145597 13960 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.145789 13960 ts_tablet_manager.cc:1398] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.159984 14004 raft_consensus.cc:482] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.160138 14004 raft_consensus.cc:504] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.160800 14004 leader_election.cc:288] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.161288 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:06.161348 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:06.161491 13076 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:06.161515 12946 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:06.161846 13144 leader_election.cc:302] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.162056 14004 raft_consensus.cc:2781] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.162120 14004 raft_consensus.cc:482] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.162166 14004 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.163136 14004 raft_consensus.cc:504] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.163456 14004 leader_election.cc:288] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.163806 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:06.163810 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:06.163918 12946 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.163923 13076 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.164629 12946 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:06.164705 13076 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:06.164909 13144 leader_election.cc:302] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.165058 14004 raft_consensus.cc:2781] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.165140 14004 raft_consensus.cc:686] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:06.165347 14004 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.167019 12667 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:06.314314 13896 raft_consensus.cc:482] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.314453 13896 raft_consensus.cc:504] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.314863 13896 leader_election.cc:288] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.315222 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b094d04434d24923806b0e9e209c3fca" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:06.315371 12813 raft_consensus.cc:2454] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:06.315661 13012 leader_election.cc:302] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.315825 13896 raft_consensus.cc:2781] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.315876 13896 raft_consensus.cc:482] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.315909 13896 raft_consensus.cc:3037] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.316701 13896 raft_consensus.cc:504] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.317011 13896 leader_election.cc:288] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.317304 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b094d04434d24923806b0e9e209c3fca" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:06.317442 12813 raft_consensus.cc:3037] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.318205 12813 raft_consensus.cc:2454] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:06.318459 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b094d04434d24923806b0e9e209c3fca" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:06.318547 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b094d04434d24923806b0e9e209c3fca" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:06.318634 13012 leader_election.cc:302] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.318667 12946 raft_consensus.cc:2454] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:06.318853 13896 raft_consensus.cc:2781] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.319016 13896 raft_consensus.cc:686] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:06.319169 13896 consensus_queue.cc:237] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.320617 12667 catalog_manager.cc:5484] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:06.335196 14004 raft_consensus.cc:482] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.335348 14004 raft_consensus.cc:504] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.335754 14004 leader_election.cc:288] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:06.336117 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:06.336175 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:06.336282 12813 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:06.336282 13076 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:06.336635 13143 leader_election.cc:302] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.336846 14004 raft_consensus.cc:2781] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.336892 14004 raft_consensus.cc:482] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.336928 14004 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.337714 14004 raft_consensus.cc:504] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.338034 14004 leader_election.cc:288] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:06.338349 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:06.338364 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:06.338464 13076 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.338470 12813 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.339169 13076 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:06.339222 12813 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:06.339442 13141 leader_election.cc:302] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.339605 14004 raft_consensus.cc:2781] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.339927 14004 raft_consensus.cc:686] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:06.340075 14004 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.341630 12667 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
W20240503 08:00:06.470386 14032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:06.470659 14032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:06.473749 14032 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:00:06.573498 12945 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.574277 12813 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.574677 12945 raft_consensus.cc:3037] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.574652 14029 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.575085 14011 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.575608 12945 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.575628 13076 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.575963 14029 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.580708 14011 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.587754 12815 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.588129 13806 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.592255 13077 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.592762 14011 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.449 ms
time per row: 0.0617243 ms
I20240503 08:00:06.637993 12673 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:06.643756 13056 tablet_service.cc:1467] Processing CreateTablet for tablet a60b868dcc504bb29d08e2752c8dd44d (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.643889 12926 tablet_service.cc:1467] Processing CreateTablet for tablet a60b868dcc504bb29d08e2752c8dd44d (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.644101 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.644121 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.644629 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.644865 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.645836 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4f2a993164fb6b1df674311698006 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.646109 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.646258 12794 tablet_service.cc:1467] Processing CreateTablet for tablet a60b868dcc504bb29d08e2752c8dd44d (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.646484 12794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.646570 14072 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:06.647397 14072 tablet_bootstrap.cc:654] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.648123 13960 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:06.648417 14072 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:06.648527 14072 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:06.648892 13960 tablet_bootstrap.cc:654] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.649025 14074 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:06.649005 14072 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.649120 14072 raft_consensus.cc:374] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.649153 14072 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.649289 14072 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.649595 14072 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.649539 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.649768 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.649892 14074 tablet_bootstrap.cc:654] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.650696 13960 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:06.650789 13960 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:06.650856 14074 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:06.650978 14074 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:06.651286 13187 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4f2a993164fb6b1df674311698006 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.651497 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.651428 14074 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.651481 13960 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.651543 14074 raft_consensus.cc:374] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.651572 13960 raft_consensus.cc:374] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.651576 14074 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.651605 13960 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.651700 14074 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.651733 13960 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.652012 13960 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.652184 13960 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:06.651996 14074 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.652520 14077 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:06.652666 14074 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:06.653319 14077 tablet_bootstrap.cc:654] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.653394 12793 tablet_service.cc:1467] Processing CreateTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.653575 14074 tablet_bootstrap.cc:654] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.653607 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.655059 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4f2a993164fb6b1df674311698006 (DEFAULT_TABLE table=rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:06.655285 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.657040 13960 tablet_bootstrap.cc:654] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.657166 14072 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:06.658061 14077 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:06.658144 14077 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:06.658581 14077 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.658695 14077 raft_consensus.cc:374] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.658730 14077 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.658772 14072 tablet_bootstrap.cc:654] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.658843 14077 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.659126 14077 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:06.659286 13960 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:06.659286 14077 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:06.659363 13960 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:06.659519 14074 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:06.659592 14074 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:06.659823 13960 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.659916 13960 raft_consensus.cc:374] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.659945 13960 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.659998 14074 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.660053 13960 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.660115 14077 tablet_bootstrap.cc:654] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.660109 14074 raft_consensus.cc:374] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.660145 14074 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.660328 13960 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.660414 14074 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.660787 14074 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:06.660955 14074 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:06.661976 14074 tablet_bootstrap.cc:654] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:06.662825 14077 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:06.662909 14077 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:06.663619 14074 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:06.663718 14074 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:06.663865 14072 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:06.664072 14072 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:00:06.664134 14077 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.664167 14074 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.664235 14077 raft_consensus.cc:374] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.664266 14077 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.664274 14074 raft_consensus.cc:374] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.664309 14074 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.664451 14074 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.664718 14074 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.664824 14077 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.665058 14072 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.665150 14077 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:06.665155 14072 raft_consensus.cc:374] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:06.665184 14072 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:06.665290 14072 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.665567 14072 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:06.713061 13943 raft_consensus.cc:482] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.713187 13943 raft_consensus.cc:504] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.713608 13943 leader_election.cc:288] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:06.714040 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:06.714128 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:06.714195 13077 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:06.714244 13208 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:06.714560 12748 leader_election.cc:302] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.714737 13943 raft_consensus.cc:2781] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.714808 13943 raft_consensus.cc:482] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.714840 13943 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.715919 13943 raft_consensus.cc:504] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.716261 13943 leader_election.cc:288] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:06.716497 14029 raft_consensus.cc:482] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.716593 14029 raft_consensus.cc:504] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.716681 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:06.716676 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:06.716796 13208 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.716795 13077 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.716938 14029 leader_election.cc:288] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.717206 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60b868dcc504bb29d08e2752c8dd44d" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:06.717304 12813 raft_consensus.cc:2454] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:06.717540 13012 leader_election.cc:302] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.717679 13077 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:06.717703 13208 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:06.717700 14029 raft_consensus.cc:2781] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.717764 14029 raft_consensus.cc:482] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.717797 14029 raft_consensus.cc:3037] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.718009 12748 leader_election.cc:302] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:06.718169 13943 raft_consensus.cc:2781] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.718353 13943 raft_consensus.cc:686] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:06.718494 13943 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.718600 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60b868dcc504bb29d08e2752c8dd44d" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:06.718660 14029 raft_consensus.cc:504] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.718734 12945 raft_consensus.cc:2454] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:06.718991 14029 leader_election.cc:288] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:06.719415 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60b868dcc504bb29d08e2752c8dd44d" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:06.719511 12813 raft_consensus.cc:3037] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.719524 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60b868dcc504bb29d08e2752c8dd44d" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:06.719619 12945 raft_consensus.cc:3037] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.720364 12945 raft_consensus.cc:2454] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:06.720358 12813 raft_consensus.cc:2454] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:06.720363 12668 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:06.720634 13012 leader_election.cc:302] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.720785 14029 raft_consensus.cc:2781] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.720847 14029 raft_consensus.cc:686] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:06.720952 14029 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:06.722651 12667 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:06.768863 14007 raft_consensus.cc:482] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:06.769021 14007 raft_consensus.cc:504] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.769562 14007 leader_election.cc:288] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:06.770051 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:06.770215 12813 raft_consensus.cc:2454] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:06.770488 12881 leader_election.cc:302] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.770673 14007 raft_consensus.cc:2781] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:06.770740 14007 raft_consensus.cc:482] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:06.770771 14007 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.771941 14007 raft_consensus.cc:504] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.772333 14007 leader_election.cc:288] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:06.772706 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:06.772835 12813 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.773700 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:06.773753 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:06.773766 12813 raft_consensus.cc:2454] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 1.
I20240503 08:00:06.773828 13208 raft_consensus.cc:2454] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:06.774089 12881 leader_election.cc:302] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:06.774300 14007 raft_consensus.cc:2781] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:06.774487 14007 raft_consensus.cc:686] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:06.774637 14007 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:06.776366 12667 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
W20240503 08:00:06.906198 14090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:06.906508 14090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:06.909546 14090 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:00:06.984128 13208 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:06.985232 13208 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.985652 14007 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.987473 13077 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.989809 12945 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.990001 12813 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:06.990398 14029 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.990522 13896 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:06.990494 14084 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.005203 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.005633 13943 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.013983 12813 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.014429 14007 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.030179 13937 mvcc.cc:205] Tried to move back new op lower bound from 7023506255799226368 to 7023506254950268928. Current Snapshot: MvccSnapshot[applied={T|T < 7023506255799226368}]
INSERT report
rows total: 2000
time total: 102.416 ms
time per row: 0.051208 ms
I20240503 08:00:07.052353 12670 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:07.057613 12926 tablet_service.cc:1467] Processing CreateTablet for tablet d44b2c67014742af9a65a3576f509702 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.057703 13187 tablet_service.cc:1467] Processing CreateTablet for tablet d44b2c67014742af9a65a3576f509702 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.057703 13057 tablet_service.cc:1467] Processing CreateTablet for tablet d44b2c67014742af9a65a3576f509702 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.057930 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44b2c67014742af9a65a3576f509702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.057973 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44b2c67014742af9a65a3576f509702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.057991 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44b2c67014742af9a65a3576f509702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.059273 12793 tablet_service.cc:1467] Processing CreateTablet for tablet f1151731adc9407b86d143d6d41f69b4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.059548 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1151731adc9407b86d143d6d41f69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.059864 14072 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:07.059973 13960 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:07.060690 13960 tablet_bootstrap.cc:654] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.060694 14072 tablet_bootstrap.cc:654] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.061182 14077 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:07.061641 14074 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:07.061837 14072 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:07.061903 14072 ts_tablet_manager.cc:1367] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:07.062155 14077 tablet_bootstrap.cc:654] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.062305 14072 raft_consensus.cc:348] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.062404 14072 raft_consensus.cc:374] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.062434 14072 raft_consensus.cc:729] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.062450 14074 tablet_bootstrap.cc:654] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.062553 14072 consensus_queue.cc:260] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.062839 14072 ts_tablet_manager.cc:1398] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:07.063426 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 076781af2ac14ddc87d31e3ec77db08f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.063606 13187 tablet_service.cc:1467] Processing CreateTablet for tablet 076781af2ac14ddc87d31e3ec77db08f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.063660 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.063807 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.064031 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 076781af2ac14ddc87d31e3ec77db08f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.064221 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.064967 13056 tablet_service.cc:1467] Processing CreateTablet for tablet f1151731adc9407b86d143d6d41f69b4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.065171 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1151731adc9407b86d143d6d41f69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.065905 14072 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:07.066056 12924 tablet_service.cc:1467] Processing CreateTablet for tablet f1151731adc9407b86d143d6d41f69b4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.066234 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1151731adc9407b86d143d6d41f69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.066237 13960 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:07.066311 13960 ts_tablet_manager.cc:1367] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:07.066730 13960 raft_consensus.cc:348] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.066821 13960 raft_consensus.cc:374] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.066848 13960 raft_consensus.cc:729] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.066895 14072 tablet_bootstrap.cc:654] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.067095 13960 consensus_queue.cc:260] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.067363 13960 ts_tablet_manager.cc:1398] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:07.067523 13960 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:07.068329 13960 tablet_bootstrap.cc:654] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.071228 14077 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:07.071309 14077 ts_tablet_manager.cc:1367] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:00:07.071331 13960 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:07.071400 13960 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:00:07.071565 14074 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:07.071647 14074 ts_tablet_manager.cc:1367] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:00:07.071750 13960 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.071734 14077 raft_consensus.cc:348] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.071826 13960 raft_consensus.cc:374] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.071830 14077 raft_consensus.cc:374] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.071854 13960 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.071859 14077 raft_consensus.cc:729] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.071964 14077 consensus_queue.cc:260] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.072079 14074 raft_consensus.cc:348] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.072181 14074 raft_consensus.cc:374] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.072213 14074 raft_consensus.cc:729] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.072223 14077 ts_tablet_manager.cc:1398] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:07.072340 14074 consensus_queue.cc:260] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.072377 14077 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:07.072571 14074 ts_tablet_manager.cc:1398] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.072844 13960 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.073146 13960 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:07.073179 14077 tablet_bootstrap.cc:654] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.073412 13960 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:07.074260 13960 tablet_bootstrap.cc:654] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.074838 14077 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:07.074946 14077 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:07.075342 14077 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.075444 14077 raft_consensus.cc:374] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.075474 14077 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.075573 14077 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.075640 14072 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:07.075708 14072 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:00:07.075834 14077 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.076100 14072 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.076175 14072 raft_consensus.cc:374] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.076200 14072 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.076306 14072 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.076575 14072 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:07.076722 14072 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:07.077531 14072 tablet_bootstrap.cc:654] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.078552 13960 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:07.078637 13960 ts_tablet_manager.cc:1367] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:00:07.078727 14072 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:07.078801 14072 ts_tablet_manager.cc:1367] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:07.079082 13960 raft_consensus.cc:348] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.079178 13960 raft_consensus.cc:374] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.079218 13960 raft_consensus.cc:729] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.079192 14072 raft_consensus.cc:348] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.079268 14072 raft_consensus.cc:374] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.079305 14072 raft_consensus.cc:729] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.079329 13960 consensus_queue.cc:260] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.079406 14072 consensus_queue.cc:260] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.079593 13960 ts_tablet_manager.cc:1398] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.080320 14072 ts_tablet_manager.cc:1398] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.123522 14011 raft_consensus.cc:482] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.123651 14011 raft_consensus.cc:504] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.124063 14011 leader_election.cc:288] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:07.124337 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44b2c67014742af9a65a3576f509702" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:07.124490 12945 raft_consensus.cc:2454] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:07.124471 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44b2c67014742af9a65a3576f509702" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:07.124605 13077 raft_consensus.cc:2454] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:07.124789 13144 leader_election.cc:302] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.124953 14011 raft_consensus.cc:2781] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.125015 14011 raft_consensus.cc:482] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.125046 14011 raft_consensus.cc:3037] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.125859 14011 raft_consensus.cc:504] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.126178 14011 leader_election.cc:288] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:07.126458 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44b2c67014742af9a65a3576f509702" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:07.126487 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44b2c67014742af9a65a3576f509702" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:07.126559 12945 raft_consensus.cc:3037] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.126585 13077 raft_consensus.cc:3037] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.127286 13077 raft_consensus.cc:2454] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:07.127297 12945 raft_consensus.cc:2454] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:07.127562 13141 leader_election.cc:302] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.127736 14011 raft_consensus.cc:2781] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.127817 14011 raft_consensus.cc:686] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:07.127961 14011 consensus_queue.cc:237] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.129529 12673 catalog_manager.cc:5484] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:07.136236 14127 raft_consensus.cc:482] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.136322 14127 raft_consensus.cc:504] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.136622 14127 leader_election.cc:288] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:07.136931 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076781af2ac14ddc87d31e3ec77db08f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:07.136924 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076781af2ac14ddc87d31e3ec77db08f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:07.137040 13207 raft_consensus.cc:2454] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:07.137045 12945 raft_consensus.cc:2454] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:07.137277 13010 leader_election.cc:302] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.137437 14127 raft_consensus.cc:2781] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.137486 14127 raft_consensus.cc:482] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.137521 14127 raft_consensus.cc:3037] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.138278 14127 raft_consensus.cc:504] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.138548 14127 leader_election.cc:288] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:07.138844 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076781af2ac14ddc87d31e3ec77db08f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:07.138949 13207 raft_consensus.cc:3037] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.139256 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076781af2ac14ddc87d31e3ec77db08f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:07.139343 12945 raft_consensus.cc:3037] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.139739 13207 raft_consensus.cc:2454] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:07.139976 13010 leader_election.cc:302] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.140020 12945 raft_consensus.cc:2454] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:07.140123 14127 raft_consensus.cc:2781] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.140197 14127 raft_consensus.cc:686] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:07.140323 14127 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:07.141948 12673 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:07.402498 14127 raft_consensus.cc:482] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.402634 14127 raft_consensus.cc:504] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.403097 14127 leader_election.cc:288] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.403548 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1151731adc9407b86d143d6d41f69b4" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:07.403595 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1151731adc9407b86d143d6d41f69b4" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:07.403697 12945 raft_consensus.cc:2454] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:07.403705 12813 raft_consensus.cc:2454] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:07.404016 13013 leader_election.cc:302] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:07.404210 14127 raft_consensus.cc:2781] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.404276 14127 raft_consensus.cc:482] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.404320 14127 raft_consensus.cc:3037] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.405117 14127 raft_consensus.cc:504] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.405437 14127 leader_election.cc:288] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.405746 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1151731adc9407b86d143d6d41f69b4" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:07.405854 12813 raft_consensus.cc:3037] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.405822 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1151731adc9407b86d143d6d41f69b4" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:07.405943 12945 raft_consensus.cc:3037] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.406582 12813 raft_consensus.cc:2454] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:07.406774 12945 raft_consensus.cc:2454] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:07.406879 13012 leader_election.cc:302] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:07.407110 14127 raft_consensus.cc:2781] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.407207 14127 raft_consensus.cc:686] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:07.407361 14127 consensus_queue.cc:237] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:07.409147 12673 catalog_manager.cc:5484] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
W20240503 08:00:07.543203 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:07.543506 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:07.546640 14144 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:00:07.607829 14127 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.612229 14127 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.638443 12945 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.638649 13077 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.638861 14011 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.639019 12813 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.639565 12945 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:07.639580 14127 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.639678 14011 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.640015 14127 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:07.646909 14168 mvcc.cc:205] Tried to move back new op lower bound from 7023506258482622464 to 7023506256397135872. Current Snapshot: MvccSnapshot[applied={T|T < 7023506258482622464}]
I20240503 08:00:07.654212 14186 mvcc.cc:205] Tried to move back new op lower bound from 7023506258483372032 to 7023506257541914624. Current Snapshot: MvccSnapshot[applied={T|T < 7023506258483372032}]
INSERT report
rows total: 2000
time total: 114.668 ms
time per row: 0.0573342 ms
I20240503 08:00:07.701431 12672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:07.706678 12793 tablet_service.cc:1467] Processing CreateTablet for tablet 047aba81fad543ce8b562a184746b163 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.707029 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047aba81fad543ce8b562a184746b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.706980 13187 tablet_service.cc:1467] Processing CreateTablet for tablet 047aba81fad543ce8b562a184746b163 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.707226 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047aba81fad543ce8b562a184746b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.707836 13056 tablet_service.cc:1467] Processing CreateTablet for tablet d9a2990625964c0e8b79780890b0e6c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.708117 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a2990625964c0e8b79780890b0e6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.709195 14202 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:07.709426 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 047aba81fad543ce8b562a184746b163 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.709683 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047aba81fad543ce8b562a184746b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.710078 14202 tablet_bootstrap.cc:654] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.710060 13188 tablet_service.cc:1467] Processing CreateTablet for tablet d9a2990625964c0e8b79780890b0e6c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.710306 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a2990625964c0e8b79780890b0e6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.710527 14203 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:07.711289 14203 tablet_bootstrap.cc:654] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.711948 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.712172 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.713446 14202 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:07.713517 14202 ts_tablet_manager.cc:1367] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:07.713894 14202 raft_consensus.cc:348] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.713992 14202 raft_consensus.cc:374] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.714022 14202 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.714148 14202 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.714416 14202 ts_tablet_manager.cc:1398] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.714579 14202 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:07.714751 12793 tablet_service.cc:1467] Processing CreateTablet for tablet d9a2990625964c0e8b79780890b0e6c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.714808 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.715057 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a2990625964c0e8b79780890b0e6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.715248 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (DEFAULT_TABLE table=rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:07.715390 14202 tablet_bootstrap.cc:654] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.715479 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.716975 14207 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:07.717103 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.717883 14207 tablet_bootstrap.cc:654] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.718860 14207 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:07.719085 14207 ts_tablet_manager.cc:1367] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:07.719524 14204 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:07.719496 14207 raft_consensus.cc:348] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.719691 14202 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:07.719703 14207 raft_consensus.cc:374] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.719769 14202 ts_tablet_manager.cc:1367] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:07.719839 14207 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.720027 14207 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.720182 14202 raft_consensus.cc:348] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.720288 14202 raft_consensus.cc:374] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.720325 14204 tablet_bootstrap.cc:654] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.720328 14202 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.720431 14202 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.720497 14207 ts_tablet_manager.cc:1398] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.720702 14202 ts_tablet_manager.cc:1398] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.720849 14202 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:07.721014 14207 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:07.721601 14202 tablet_bootstrap.cc:654] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.721817 14207 tablet_bootstrap.cc:654] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.722522 14202 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:07.722605 14202 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:07.722749 14207 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:07.722816 14207 ts_tablet_manager.cc:1367] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:07.723016 14202 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.723104 14202 raft_consensus.cc:374] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.723134 14202 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.723230 14202 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.723259 14207 raft_consensus.cc:348] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.723346 14207 raft_consensus.cc:374] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.723377 14207 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.723456 14202 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.723479 14207 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.723731 14207 ts_tablet_manager.cc:1398] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.723884 14207 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:07.724637 14207 tablet_bootstrap.cc:654] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.726325 14207 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:07.726389 14207 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:07.726739 14207 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.726832 14207 raft_consensus.cc:374] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.726868 14207 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.727005 14207 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.727234 14207 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.730978 14203 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:07.731076 14203 ts_tablet_manager.cc:1367] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 08:00:07.731460 14203 raft_consensus.cc:348] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.731549 14203 raft_consensus.cc:374] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.731580 14203 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.731691 14203 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.731925 14203 ts_tablet_manager.cc:1398] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:07.732602 14204 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:07.732693 14204 ts_tablet_manager.cc:1367] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 08:00:07.733129 14204 raft_consensus.cc:348] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.733227 14204 raft_consensus.cc:374] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.733261 14204 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.733373 14204 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.733708 14204 ts_tablet_manager.cc:1398] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.733845 14204 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:07.734679 14204 tablet_bootstrap.cc:654] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.736346 14204 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:07.736428 14204 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:07.736820 14204 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.736915 14204 raft_consensus.cc:374] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.736944 14204 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.737056 14204 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.737322 14204 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:07.737663 13943 raft_consensus.cc:482] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.737736 13943 raft_consensus.cc:504] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.738045 13943 leader_election.cc:288] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.738447 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:07.738575 13207 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:07.738592 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:07.738696 12946 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:07.738854 12748 leader_election.cc:302] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.739117 13943 raft_consensus.cc:2781] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.739169 13943 raft_consensus.cc:482] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.739193 13943 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.739887 13943 raft_consensus.cc:504] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.740180 13943 leader_election.cc:288] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.740455 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:07.740474 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:07.740542 13207 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.740559 12946 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.741166 13207 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:07.741205 12946 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:07.741420 12748 leader_election.cc:302] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.741608 13943 raft_consensus.cc:2781] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.741768 13943 raft_consensus.cc:686] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:07.741883 13943 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.743211 12672 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:07.862630 14011 raft_consensus.cc:482] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.862751 14011 raft_consensus.cc:504] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.863134 14011 leader_election.cc:288] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.863489 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:07.863518 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:07.863637 12813 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:07.863651 12946 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:07.863879 13144 leader_election.cc:302] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.864043 14011 raft_consensus.cc:2781] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.864096 14011 raft_consensus.cc:482] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.864121 14011 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.864882 14011 raft_consensus.cc:504] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.865164 14011 leader_election.cc:288] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:07.865427 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:07.865437 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:07.865515 12946 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.865527 12813 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.866164 12946 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:07.866201 12813 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:07.866412 13144 leader_election.cc:302] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.866588 14011 raft_consensus.cc:2781] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.866662 14011 raft_consensus.cc:686] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:07.866791 14011 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:07.868466 12672 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:07.887706 13943 raft_consensus.cc:482] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:07.887806 13943 raft_consensus.cc:504] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.888146 13943 leader_election.cc:288] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:07.888581 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:07.888613 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:07.888720 13077 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:07.888743 13207 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:07.888980 12748 leader_election.cc:302] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.889142 13943 raft_consensus.cc:2781] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:07.889195 13943 raft_consensus.cc:482] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:07.889231 13943 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.889948 13943 raft_consensus.cc:504] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.890244 13943 leader_election.cc:288] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:07.890575 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:07.890693 13077 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.890641 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:07.890765 13207 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.891506 13207 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:07.891566 13077 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:07.891814 12748 leader_election.cc:302] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:07.891955 13943 raft_consensus.cc:2781] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.892032 13943 raft_consensus.cc:686] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:07.892185 13943 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:07.896014 12672 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
W20240503 08:00:08.003425 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.003661 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.006815 14219 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:00:08.089560 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.089910 13207 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.090812 12813 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.091312 14215 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.091646 13077 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.091779 14180 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.092402 13943 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.092722 13943 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.092795 13207 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.093133 12946 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.095157 14215 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.096990 14011 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.113687 14184 mvcc.cc:205] Tried to move back new op lower bound from 7023506260322320384 to 7023506258911653888. Current Snapshot: MvccSnapshot[applied={T|T < 7023506260322320384}]
INSERT report
rows total: 2000
time total: 105.395 ms
time per row: 0.0526976 ms
I20240503 08:00:08.151273 12672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:08.156893 12795 tablet_service.cc:1467] Processing CreateTablet for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.157220 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdde0a31f7d4fd0bf20c894f215e3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.157538 13056 tablet_service.cc:1467] Processing CreateTablet for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.157819 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdde0a31f7d4fd0bf20c894f215e3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.159154 14207 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:08.159619 14203 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:08.160068 14207 tablet_bootstrap.cc:654] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.160203 12924 tablet_service.cc:1467] Processing CreateTablet for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.160447 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdde0a31f7d4fd0bf20c894f215e3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.160475 14203 tablet_bootstrap.cc:654] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.160926 13188 tablet_service.cc:1467] Processing CreateTablet for tablet a45748a98c3447d3aa2e8de94981a745 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.161073 12926 tablet_service.cc:1467] Processing CreateTablet for tablet a45748a98c3447d3aa2e8de94981a745 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.161185 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.161173 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.161270 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.161363 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.162523 13057 tablet_service.cc:1467] Processing CreateTablet for tablet a45748a98c3447d3aa2e8de94981a745 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.162740 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.163237 14204 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:08.163280 14202 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:08.163885 14207 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:08.163960 14207 ts_tablet_manager.cc:1367] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:08.164034 14202 tablet_bootstrap.cc:654] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.164070 14204 tablet_bootstrap.cc:654] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.164448 14207 raft_consensus.cc:348] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.164557 14207 raft_consensus.cc:374] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.164590 14207 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.160926 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.164717 14207 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.164780 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.164932 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.164999 14207 ts_tablet_manager.cc:1398] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.165163 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.166534 14202 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:08.166615 14202 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:08.167073 14202 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.167188 14202 raft_consensus.cc:374] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.167229 14202 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.167331 14203 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:08.167346 14202 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.167412 14203 ts_tablet_manager.cc:1367] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:08.167618 14202 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.167786 14202 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:08.167920 14203 raft_consensus.cc:348] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.168020 14203 raft_consensus.cc:374] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.168058 14203 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.168174 14203 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.168447 14203 ts_tablet_manager.cc:1398] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.168975 14207 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:08.169690 14203 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:08.169834 14207 tablet_bootstrap.cc:654] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.170507 14203 tablet_bootstrap.cc:654] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.171720 14207 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:08.171727 14204 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:08.171802 14207 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:08.171802 14204 ts_tablet_manager.cc:1367] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:00:08.172248 14204 raft_consensus.cc:348] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.172248 14207 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.172353 14207 raft_consensus.cc:374] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.172353 14204 raft_consensus.cc:374] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.172384 14207 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.172384 14204 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.172488 14207 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.172497 14204 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.172748 14207 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.172761 14204 ts_tablet_manager.cc:1398] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.172907 14204 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:08.173709 14204 tablet_bootstrap.cc:654] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.168773 14202 tablet_bootstrap.cc:654] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.174899 14203 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:08.175007 14203 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:08.175453 14204 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:08.175423 14203 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.175516 14203 raft_consensus.cc:374] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.175535 14204 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:08.175554 14203 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.175673 14203 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.175936 14203 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:08.175936 14204 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.176023 14204 raft_consensus.cc:374] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.176072 14204 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.176105 14203 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:08.176183 14204 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.176456 14204 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.176940 14203 tablet_bootstrap.cc:654] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.177573 14202 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:08.177687 14202 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:00:08.177964 14203 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:08.178035 14203 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:08.178133 14202 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.178241 14202 raft_consensus.cc:374] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.178279 14202 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.178393 14202 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.178418 14203 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.178601 14203 raft_consensus.cc:374] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.178642 14203 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.178668 14202 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:08.178786 14203 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.179072 14203 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:08.242339 14127 raft_consensus.cc:482] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.242461 14127 raft_consensus.cc:504] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.242836 14127 leader_election.cc:288] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:08.243178 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45748a98c3447d3aa2e8de94981a745" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:08.243223 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45748a98c3447d3aa2e8de94981a745" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:08.243331 13207 raft_consensus.cc:2454] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:08.243331 12945 raft_consensus.cc:2454] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:08.243589 13013 leader_election.cc:302] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:08.243760 14127 raft_consensus.cc:2781] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.243832 14127 raft_consensus.cc:482] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.243870 14127 raft_consensus.cc:3037] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.244800 14127 raft_consensus.cc:504] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.245102 14127 leader_election.cc:288] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:08.245375 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45748a98c3447d3aa2e8de94981a745" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:08.245379 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45748a98c3447d3aa2e8de94981a745" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:08.245491 13207 raft_consensus.cc:3037] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.245487 12945 raft_consensus.cc:3037] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.246390 12945 raft_consensus.cc:2454] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:08.246390 13207 raft_consensus.cc:2454] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:08.246665 13013 leader_election.cc:302] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:08.246810 14127 raft_consensus.cc:2781] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.246889 14127 raft_consensus.cc:686] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:08.247042 14127 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.248752 12668 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:00:08.277107 14088 raft_consensus.cc:482] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.277235 14088 raft_consensus.cc:504] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.277644 14088 leader_election.cc:288] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:08.278113 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:08.278090 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:08.278244 13077 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:08.278249 12813 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 0.
I20240503 08:00:08.278553 12879 leader_election.cc:302] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:08.278736 14088 raft_consensus.cc:2781] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.278793 14088 raft_consensus.cc:482] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.278820 14088 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.279589 14088 raft_consensus.cc:504] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.279886 14088 leader_election.cc:288] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:08.280223 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:08.280223 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:08.280335 12813 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.280335 13077 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.281196 13077 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 1.
I20240503 08:00:08.281196 12813 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 1.
I20240503 08:00:08.281520 12879 leader_election.cc:302] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:08.281749 14088 raft_consensus.cc:2781] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.281900 14088 raft_consensus.cc:686] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:08.282042 14088 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.283744 12668 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:08.339536 14250 raft_consensus.cc:482] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.339651 14250 raft_consensus.cc:504] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.340029 14250 leader_election.cc:288] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:08.340356 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:08.340381 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:08.340499 12813 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:08.340513 13077 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:08.340770 13143 leader_election.cc:302] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:08.340931 14250 raft_consensus.cc:2781] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.340987 14250 raft_consensus.cc:482] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.341012 14250 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.341821 14250 raft_consensus.cc:504] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.342133 14250 leader_election.cc:288] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:08.342428 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:08.342453 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:08.342527 13077 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.342558 12813 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.343281 13077 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:08.343336 12813 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:08.343523 13141 leader_election.cc:302] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:08.343672 14250 raft_consensus.cc:2781] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.343765 14250 raft_consensus.cc:686] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:08.343914 14250 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:08.345543 12668 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
W20240503 08:00:08.452903 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.453125 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.456158 14275 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:00:08.533114 13077 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.533274 12813 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.533525 14088 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.533684 12815 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.533895 14088 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.534166 13077 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.534540 14180 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.535032 14180 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.538707 12945 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.538936 13207 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:08.539125 14190 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:08.539562 14127 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.815 ms
time per row: 0.0539075 ms
I20240503 08:00:08.603165 12671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:08.608577 12795 tablet_service.cc:1467] Processing CreateTablet for tablet 7d61aa445a50417b82916019c2a4298c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.608742 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 7d61aa445a50417b82916019c2a4298c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.608901 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d61aa445a50417b82916019c2a4298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.609032 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d61aa445a50417b82916019c2a4298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.609565 12793 tablet_service.cc:1467] Processing CreateTablet for tablet 5212dff7ded143a7923bb6176466b0fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.609586 12794 tablet_service.cc:1467] Processing CreateTablet for tablet 5480e544dca3428ab4fcf190577d5331 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.609791 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.609882 12794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5480e544dca3428ab4fcf190577d5331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.611078 14204 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:08.611701 14207 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:08.611918 14204 tablet_bootstrap.cc:654] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.612483 14207 tablet_bootstrap.cc:654] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.612529 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 5480e544dca3428ab4fcf190577d5331 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.612777 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5480e544dca3428ab4fcf190577d5331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.612723 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 5212dff7ded143a7923bb6176466b0fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.612984 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.613991 14207 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:08.614076 14207 ts_tablet_manager.cc:1367] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:08.614586 14203 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:08.614509 14207 raft_consensus.cc:348] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.612529 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 7d61aa445a50417b82916019c2a4298c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.614638 14207 raft_consensus.cc:374] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.614670 14207 raft_consensus.cc:729] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.614764 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d61aa445a50417b82916019c2a4298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.614784 14207 consensus_queue.cc:260] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.615067 14207 ts_tablet_manager.cc:1398] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.615114 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 5480e544dca3428ab4fcf190577d5331 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.615231 14207 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:08.615321 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5480e544dca3428ab4fcf190577d5331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.615391 14203 tablet_bootstrap.cc:654] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.615978 14202 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:08.616169 14207 tablet_bootstrap.cc:654] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.616778 14202 tablet_bootstrap.cc:654] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.617481 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 5212dff7ded143a7923bb6176466b0fa (DEFAULT_TABLE table=rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:08.617659 14204 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:08.617699 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:08.617746 14204 ts_tablet_manager.cc:1367] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:08.618175 14204 raft_consensus.cc:348] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.618286 14204 raft_consensus.cc:374] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.618324 14204 raft_consensus.cc:729] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.618391 14207 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:08.618430 14204 consensus_queue.cc:260] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.618607 14207 ts_tablet_manager.cc:1367] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:08.618711 14204 ts_tablet_manager.cc:1398] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.619195 14207 raft_consensus.cc:348] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.619354 14207 raft_consensus.cc:374] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.619431 14207 raft_consensus.cc:729] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.619446 14202 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:08.619467 14204 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:08.619525 14202 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:08.619664 14207 consensus_queue.cc:260] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.619988 14207 ts_tablet_manager.cc:1398] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:08.619951 14202 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.620043 14202 raft_consensus.cc:374] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.620079 14202 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.620195 14202 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.620358 14207 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:08.620448 14202 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:08.620481 14204 tablet_bootstrap.cc:654] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.620599 14202 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:08.621356 14207 tablet_bootstrap.cc:654] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.621419 14202 tablet_bootstrap.cc:654] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.621793 14203 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:08.621884 14203 ts_tablet_manager.cc:1367] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:08.622327 14203 raft_consensus.cc:348] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.622424 14203 raft_consensus.cc:374] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.622437 14207 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:08.622454 14203 raft_consensus.cc:729] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.622499 14207 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:08.622565 14203 consensus_queue.cc:260] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.622818 14203 ts_tablet_manager.cc:1398] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.622875 14207 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.622979 14207 raft_consensus.cc:374] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.622995 14203 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:08.623009 14207 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.623116 14207 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.623351 14207 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:08.623787 14203 tablet_bootstrap.cc:654] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:08.624907 14203 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:08.624974 14203 ts_tablet_manager.cc:1367] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:08.625348 14203 raft_consensus.cc:348] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.625433 14203 raft_consensus.cc:374] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.625463 14203 raft_consensus.cc:729] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.625566 14203 consensus_queue.cc:260] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.625802 14203 ts_tablet_manager.cc:1398] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:08.628168 14204 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:08.628239 14204 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:08.628329 14202 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:08.628403 14202 ts_tablet_manager.cc:1367] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:08.628597 14204 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.628679 14204 raft_consensus.cc:374] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.628698 14204 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.628728 14202 raft_consensus.cc:348] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.628835 14202 raft_consensus.cc:374] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:08.628809 14204 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.628875 14202 raft_consensus.cc:729] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:08.628989 14202 consensus_queue.cc:260] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.629045 14204 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:08.629242 14202 ts_tablet_manager.cc:1398] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:08.637611 14259 raft_consensus.cc:482] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.637703 14259 raft_consensus.cc:504] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.638041 14259 leader_election.cc:288] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:08.638315 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5480e544dca3428ab4fcf190577d5331" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:08.638386 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5480e544dca3428ab4fcf190577d5331" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:08.638435 13077 raft_consensus.cc:2454] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:08.638515 13207 raft_consensus.cc:2454] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:08.638665 12748 leader_election.cc:302] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:08.638801 14259 raft_consensus.cc:2781] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.638856 14259 raft_consensus.cc:482] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.638897 14259 raft_consensus.cc:3037] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.639638 14259 raft_consensus.cc:504] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.639920 14259 leader_election.cc:288] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:08.640233 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5480e544dca3428ab4fcf190577d5331" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:08.640234 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5480e544dca3428ab4fcf190577d5331" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:08.640328 13207 raft_consensus.cc:3037] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.640328 13077 raft_consensus.cc:3037] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.640985 13077 raft_consensus.cc:2454] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:08.641023 13207 raft_consensus.cc:2454] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:08.641223 12748 leader_election.cc:302] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:08.641367 14259 raft_consensus.cc:2781] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.641511 14259 raft_consensus.cc:686] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:08.641638 14259 consensus_queue.cc:237] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:08.643151 12671 catalog_manager.cc:5484] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:08.644093 14259 raft_consensus.cc:482] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.644160 14259 raft_consensus.cc:504] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.644454 14259 leader_election.cc:288] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:08.644742 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:08.644845 13207 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:08.644802 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:08.644966 12945 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:08.645080 12748 leader_election.cc:302] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:08.645279 14259 raft_consensus.cc:2781] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.645327 14259 raft_consensus.cc:482] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.645354 14259 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.645982 14259 raft_consensus.cc:504] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.646252 14259 leader_election.cc:288] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:08.646493 13207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:08.646529 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:08.646592 13207 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.646616 12945 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.647285 13207 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:08.647294 12945 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:08.647526 12751 leader_election.cc:302] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:08.647660 14259 raft_consensus.cc:2781] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.647739 14259 raft_consensus.cc:686] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:08.647861 14259 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:08.649417 12671 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:08.942714 14127 raft_consensus.cc:482] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:08.942850 14127 raft_consensus.cc:504] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.943271 14127 leader_election.cc:288] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:08.943681 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d61aa445a50417b82916019c2a4298c" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38" is_pre_election: true
I20240503 08:00:08.943713 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d61aa445a50417b82916019c2a4298c" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:08.943851 12813 raft_consensus.cc:2454] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:08.943861 12945 raft_consensus.cc:2454] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:08.944170 13012 leader_election.cc:302] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:08.944378 14127 raft_consensus.cc:2781] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:08.944437 14127 raft_consensus.cc:482] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:08.944478 14127 raft_consensus.cc:3037] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.945276 14127 raft_consensus.cc:504] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.945613 14127 leader_election.cc:288] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:08.945864 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d61aa445a50417b82916019c2a4298c" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:08.945969 12815 raft_consensus.cc:3037] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.945995 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d61aa445a50417b82916019c2a4298c" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:08.946091 12945 raft_consensus.cc:3037] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:08.946902 12945 raft_consensus.cc:2454] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:08.946945 12815 raft_consensus.cc:2454] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:08.947270 13012 leader_election.cc:302] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:08.947422 14127 raft_consensus.cc:2781] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:08.947508 14127 raft_consensus.cc:686] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:08.947644 14127 consensus_queue.cc:237] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:08.949291 12671 catalog_manager.cc:5484] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
W20240503 08:00:09.087586 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:09.087800 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:09.091114 14331 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:09.094215 14259 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.097707 14259 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240503 08:00:09.169831 12815 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.170234 14127 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.170472 12945 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.170989 13207 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.174031 14127 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.174710 14327 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.177088 12945 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.179049 14327 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 92.8 ms
time per row: 0.0464 ms
I20240503 08:00:09.227389 12671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:09.232762 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.232836 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.233002 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 1c84f7a57e0945d9a25a70798f054bd1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.233093 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.233091 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.233251 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c84f7a57e0945d9a25a70798f054bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.235271 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 1c84f7a57e0945d9a25a70798f054bd1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.235441 14391 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:09.235538 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c84f7a57e0945d9a25a70798f054bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.235623 13057 tablet_service.cc:1467] Processing CreateTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.235810 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.236167 14391 tablet_bootstrap.cc:654] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.236972 14391 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:09.237030 14391 ts_tablet_manager.cc:1367] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:09.237339 14391 raft_consensus.cc:348] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.237439 14391 raft_consensus.cc:374] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.237470 14391 raft_consensus.cc:729] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.237628 14391 consensus_queue.cc:260] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.237730 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.237960 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.239156 14391 ts_tablet_manager.cc:1398] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:09.239429 14393 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:09.240391 12793 tablet_service.cc:1467] Processing CreateTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.240518 14393 tablet_bootstrap.cc:654] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.240659 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.241473 14393 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:09.242089 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 1c84f7a57e0945d9a25a70798f054bd1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.242307 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c84f7a57e0945d9a25a70798f054bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.242566 14393 ts_tablet_manager.cc:1367] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:09.243036 14395 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:09.243026 14393 raft_consensus.cc:348] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.243115 14393 raft_consensus.cc:374] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.243142 14393 raft_consensus.cc:729] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.243248 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e (DEFAULT_TABLE table=rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:09.243450 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:09.243814 14395 tablet_bootstrap.cc:654] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.243274 14393 consensus_queue.cc:260] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.244205 14393 ts_tablet_manager.cc:1398] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:09.244349 14393 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:09.245235 14395 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: No bootstrap required, opened a new log
I20240503 08:00:09.245311 14395 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:09.245734 14395 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.245851 14395 raft_consensus.cc:374] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.245882 14395 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.245993 14395 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.246181 14393 tablet_bootstrap.cc:654] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.246251 14395 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:09.247514 14391 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:09.248286 14393 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: No bootstrap required, opened a new log
I20240503 08:00:09.248373 14393 ts_tablet_manager.cc:1367] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:09.248379 14391 tablet_bootstrap.cc:654] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.248694 14393 raft_consensus.cc:348] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.248773 14393 raft_consensus.cc:374] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.248800 14393 raft_consensus.cc:729] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.248919 14393 consensus_queue.cc:260] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.249054 14390 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:09.249146 14393 ts_tablet_manager.cc:1398] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:09.250185 14390 tablet_bootstrap.cc:654] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.251402 14390 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:09.251420 14391 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:09.251485 14390 ts_tablet_manager.cc:1367] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:09.251497 14391 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:09.251912 14390 raft_consensus.cc:348] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.252000 14390 raft_consensus.cc:374] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.251948 14391 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.252028 14390 raft_consensus.cc:729] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.252044 14391 raft_consensus.cc:374] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.252074 14391 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.252157 14390 consensus_queue.cc:260] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.252184 14391 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.252473 14391 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:09.252617 14391 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:09.252961 14390 ts_tablet_manager.cc:1398] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:09.253108 14390 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:09.253456 14391 tablet_bootstrap.cc:654] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.253829 14390 tablet_bootstrap.cc:654] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.255437 14391 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: No bootstrap required, opened a new log
I20240503 08:00:09.255513 14391 ts_tablet_manager.cc:1367] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:09.255928 14391 raft_consensus.cc:348] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.256026 14391 raft_consensus.cc:374] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.256054 14391 raft_consensus.cc:729] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.256176 14391 consensus_queue.cc:260] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.256433 14391 ts_tablet_manager.cc:1398] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:09.256842 14390 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:09.256917 14390 ts_tablet_manager.cc:1367] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:09.257308 14390 raft_consensus.cc:348] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.257396 14390 raft_consensus.cc:374] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.257426 14390 raft_consensus.cc:729] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.257524 14390 consensus_queue.cc:260] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.257840 14390 ts_tablet_manager.cc:1398] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:09.257989 14390 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:00:09.259267 14390 tablet_bootstrap.cc:654] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:09.260624 14390 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: No bootstrap required, opened a new log
I20240503 08:00:09.260702 14390 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:09.261123 14390 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.261197 14390 raft_consensus.cc:374] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:09.261224 14390 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:00:09.261327 14390 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.261556 14390 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:09.266129 14127 raft_consensus.cc:482] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:09.266208 14127 raft_consensus.cc:504] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.266481 14127 leader_election.cc:288] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:09.266790 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
I20240503 08:00:09.266897 13208 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:09.267122 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:09.267155 13010 leader_election.cc:302] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:09.267225 12945 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 0.
I20240503 08:00:09.267302 14127 raft_consensus.cc:2781] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:09.267344 14127 raft_consensus.cc:482] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:09.267377 14127 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.267755 14250 raft_consensus.cc:482] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:09.267829 14250 raft_consensus.cc:504] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.268160 14250 leader_election.cc:288] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:09.268170 14127 raft_consensus.cc:504] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.268394 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c84f7a57e0945d9a25a70798f054bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:09.268457 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c84f7a57e0945d9a25a70798f054bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:09.268496 12945 raft_consensus.cc:2454] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:09.268555 13077 raft_consensus.cc:2454] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 0.
I20240503 08:00:09.268733 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:09.268827 12945 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.268827 13144 leader_election.cc:302] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:09.268971 14250 raft_consensus.cc:2781] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:09.269027 14250 raft_consensus.cc:482] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:09.269059 14250 raft_consensus.cc:3037] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.268472 14127 leader_election.cc:288] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:09.269266 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:09.269361 13208 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.269683 12945 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:09.269871 14250 raft_consensus.cc:504] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.269948 13013 leader_election.cc:302] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:09.270087 13208 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 1.
I20240503 08:00:09.270093 14127 raft_consensus.cc:2781] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:09.270170 14127 raft_consensus.cc:686] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:09.270205 14250 leader_election.cc:288] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:09.270298 14127 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.270525 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c84f7a57e0945d9a25a70798f054bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:09.270617 12945 raft_consensus.cc:3037] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.270642 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c84f7a57e0945d9a25a70798f054bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:09.270723 13077 raft_consensus.cc:3037] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.271359 12945 raft_consensus.cc:2454] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:09.271449 13077 raft_consensus.cc:2454] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 1.
I20240503 08:00:09.271587 13144 leader_election.cc:302] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:09.271869 14250 raft_consensus.cc:2781] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:09.271951 14250 raft_consensus.cc:686] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:09.272064 14250 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:00:09.273504 12671 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:00:09.273861 12672 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:00:09.420655 14369 raft_consensus.cc:482] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:09.420795 14369 raft_consensus.cc:504] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.421273 14369 leader_election.cc:288] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:09.421625 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b40c2136d45ab86ccf49df8d1b71e" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:00:09.421757 13077 raft_consensus.cc:2454] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:09.421706 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b40c2136d45ab86ccf49df8d1b71e" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:00:09.421864 12945 raft_consensus.cc:2454] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 0.
I20240503 08:00:09.422039 12748 leader_election.cc:302] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:09.422228 14369 raft_consensus.cc:2781] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:09.422286 14369 raft_consensus.cc:482] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:09.422322 14369 raft_consensus.cc:3037] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.423120 14369 raft_consensus.cc:504] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.423427 14369 leader_election.cc:288] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:09.423748 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b40c2136d45ab86ccf49df8d1b71e" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:09.423746 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b40c2136d45ab86ccf49df8d1b71e" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:09.423864 12945 raft_consensus.cc:3037] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.423882 13077 raft_consensus.cc:3037] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:09.424739 12945 raft_consensus.cc:2454] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:09.424739 13077 raft_consensus.cc:2454] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 1.
I20240503 08:00:09.425012 12751 leader_election.cc:302] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38; no voters:
I20240503 08:00:09.425179 14369 raft_consensus.cc:2781] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:09.425263 14369 raft_consensus.cc:686] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:09.425540 14369 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:09.427186 12672 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
W20240503 08:00:09.573415 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:09.573624 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:09.576541 14406 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 08:00:09.669154 13077 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.669481 14180 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.669488 13208 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.669786 14180 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.670106 14311 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.670939 12946 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.680522 14127 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.682262 12944 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.682439 13076 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:09.682988 14368 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:09.683724 14368 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.89 ms
time per row: 0.0659448 ms
I20240503 08:00:09.750176 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:38311
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:37099
--webserver_port=41555
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:00:09.851308 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:09.851492 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:09.851532 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:09.854461 14445 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:09.854518 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:09.854599 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:00:09.858096 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:38311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=41555
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:09.858855 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:09.859911 14445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:09.868273 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:09.868928 14445 server_base.cc:1034] running on GCE node
W20240503 08:00:09.869561 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:09.870085 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:09.870443 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:09.870990 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:09.872126 14445 hybrid_clock.cc:648] HybridClock initialized: now 1714723209872111 us; error 29 us; skew 500 ppm
I20240503 08:00:09.873888 14445 webserver.cc:460] Webserver started at http://127.0.99.133:41555/ using document root <none> and password file <none>
I20240503 08:00:09.874464 14445 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:09.874529 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:09.877945 14445 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:09.879951 14459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:09.880973 14445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:09.881067 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "82cd33f2bae4419b946095f8eb93e1e4"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:09.881476 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:09.895593 14445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:09.896118 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:09.896256 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:09.896733 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:09.897534 14445 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:09.897578 14445 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:09.897626 14445 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:09.897652 14445 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:09.906383 14445 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38311
I20240503 08:00:09.906422 14572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38311 every 8 connection(s)
I20240503 08:00:09.907441 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:00:09.915071 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 14445
I20240503 08:00:09.915467 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:39483
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:37099
--webserver_port=38591
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:09.916556 14573 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:09.916764 14573 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:09.917215 14573 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:09.918323 12668 ts_manager.cc:194] Re-registered known tserver with Master: 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:09.918975 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:37165
W20240503 08:00:10.020885 14576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.021085 14576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:10.021139 14576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.024288 14576 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:10.024354 14576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:10.024433 14576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 08:00:10.027954 14576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:39483
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38591
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.14576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:10.028776 14576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:10.029827 14576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:10.038133 14582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.038681 14581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.039310 14584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:10.039885 14576 server_base.cc:1034] running on GCE node
I20240503 08:00:10.040256 14576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:10.040778 14576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:10.041932 14576 hybrid_clock.cc:648] HybridClock initialized: now 1714723210041922 us; error 22 us; skew 500 ppm
I20240503 08:00:10.043733 14576 webserver.cc:460] Webserver started at http://127.0.99.134:38591/ using document root <none> and password file <none>
I20240503 08:00:10.044257 14576 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:10.044327 14576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:10.047535 14576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:10.049304 14590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.050442 14576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:10.050544 14576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8398e6f89bf5419d81ed5bed991da0cf"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:10.050980 14576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:10.076511 14576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:10.077142 14576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:10.077390 14576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:10.077972 14576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:10.078912 14576 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:10.078992 14576 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.079037 14576 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:10.079064 14576 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.088249 14576 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39483
I20240503 08:00:10.088280 14703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39483 every 8 connection(s)
I20240503 08:00:10.089430 14576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:00:10.091956 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 14576
I20240503 08:00:10.092326 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:37289
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:37099
--webserver_port=33423
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:10.098814 14704 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:10.099076 14704 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:10.099610 14704 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:10.100584 12668 ts_manager.cc:194] Re-registered known tserver with Master: 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:10.101116 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34719
W20240503 08:00:10.199278 14707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.199460 14707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:10.199506 14707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.202473 14707 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:10.202536 14707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:10.202613 14707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 08:00:10.206223 14707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:37289
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=33423
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.14707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:10.207051 14707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:10.208230 14707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:10.216913 14715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.217418 14712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.217986 14713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:10.218580 14707 server_base.cc:1034] running on GCE node
I20240503 08:00:10.218952 14707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:10.219488 14707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:10.220628 14707 hybrid_clock.cc:648] HybridClock initialized: now 1714723210220609 us; error 35 us; skew 500 ppm
I20240503 08:00:10.222304 14707 webserver.cc:460] Webserver started at http://127.0.99.135:33423/ using document root <none> and password file <none>
I20240503 08:00:10.222810 14707 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:10.222880 14707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:10.226449 14707 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:10.228438 14721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.229360 14707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:10.229450 14707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b052e89b63c24258872c832a6a0d4d4d"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:10.229836 14707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:10.253525 14707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:10.254132 14707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:10.254290 14707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:10.254787 14707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:10.255686 14707 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:10.255733 14707 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.255781 14707 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:10.255811 14707 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.264322 14707 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37289
I20240503 08:00:10.264380 14834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37289 every 8 connection(s)
I20240503 08:00:10.265247 14707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:00:10.267267 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 14707
I20240503 08:00:10.267621 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:35323
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:37099
--webserver_port=39099
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:00:10.275034 14835 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:10.275259 14835 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:10.275756 14835 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:10.276715 12668 ts_manager.cc:194] Re-registered known tserver with Master: b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:10.277269 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:47247
W20240503 08:00:10.371140 14838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.371316 14838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:10.371358 14838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.374243 14838 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:00:10.374300 14838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:10.374361 14838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:10.378067 14838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:35323
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=39099
--tserver_master_addrs=127.0.99.190:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.14838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:10.378810 14838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:10.379837 14838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:10.387969 14843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.388590 14844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:10.389077 14846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:10.389678 14838 server_base.cc:1034] running on GCE node
I20240503 08:00:10.390086 14838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:10.390656 14838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:10.391813 14838 hybrid_clock.cc:648] HybridClock initialized: now 1714723210391788 us; error 35 us; skew 500 ppm
I20240503 08:00:10.393771 14838 webserver.cc:460] Webserver started at http://127.0.99.136:39099/ using document root <none> and password file <none>
I20240503 08:00:10.394266 14838 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:10.394325 14838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:10.397493 14838 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:10.399313 14852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.400327 14838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:10.400431 14838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "eae9f53215a34bc88abe7b3c1c223319"
format_stamp: "Formatted at 2024-05-03 08:00:00 on dist-test-slave-11df"
I20240503 08:00:10.400832 14838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:10.426550 14838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:10.427182 14838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:10.427331 14838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:10.427850 14838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:10.428696 14838 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:10.428750 14838 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.428803 14838 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:10.428828 14838 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:10.438019 14838 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35323
I20240503 08:00:10.438057 14965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35323 every 8 connection(s)
I20240503 08:00:10.439083 14838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:10.443224 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 14838
I20240503 08:00:10.450876 14966 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37099
I20240503 08:00:10.451115 14966 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:10.451658 14966 heartbeater.cc:511] Master 127.0.99.190:37099 requested a full tablet report, sending...
I20240503 08:00:10.452618 12668 ts_manager.cc:194] Re-registered known tserver with Master: eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:10.453328 12668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:56913
I20240503 08:00:10.840600 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:10.841841 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:10.854236 12926 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:10.865272 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.865455 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.879675 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.889367 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.899361 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:10.918569 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.923220 14573 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:10.937978 12926 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:10.938127 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:10.944213 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.948093 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.968065 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:10.975409 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.980500 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.019809 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.024546 12926 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.024546 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.026890 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.032549 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.040009 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.046465 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.057729 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.062629 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.071451 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.071451 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.072862 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.073792 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.074215 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.096017 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.102861 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.103353 14704 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:11.103448 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.117208 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.138657 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.145227 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.153373 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.169493 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.171854 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.177119 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.279070 14835 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:11.340234 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.350987 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.365531 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.367480 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.382683 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.394383 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.402802 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.413307 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.423491 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.426192 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.430258 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.430786 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.444244 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.445012 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.455770 14966 heartbeater.cc:503] Master 127.0.99.190:37099 was elected leader, sending a full tablet report...
I20240503 08:00:11.462709 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.469725 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.470549 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.470774 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.473766 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.483613 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.493384 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.504416 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.519596 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.522670 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.543037 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.549219 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.551524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.562613 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.564913 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.574792 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.582512 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.595432 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.602851 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.604974 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.605163 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.624246 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.642720 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.654521 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.656879 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.674115 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.830579 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.841429 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.846490 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.850163 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.867733 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.870797 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.871152 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.873605 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:11.874591 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.879266 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.886070 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.907328 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.920437 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.927069 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.929819 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.930716 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.934222 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.935431 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.938557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.947873 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.951045 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.955091 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:11.955912 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:11.963598 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.968154 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.970686 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:11.992434 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:11.994411 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.028151 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:12.032701 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.047770 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.064070 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.132737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:12.138178 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:12.140620 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.146011 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.146100 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.149272 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.150089 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:12.183436 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:12.184616 13208 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.186645 12945 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.187446 14250 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.191313 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.196082 15762 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.201598 12945 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.201855 14250 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.203999 13076 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.204643 12945 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.205706 12665 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.205935 14442 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.206825 12815 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.208114 15778 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.208988 13076 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
W20240503 08:00:12.215874 13141 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20cc823a5ce34916894cfde0a6614bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.219189 14442 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.221266 13077 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.221278 12815 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.224859 12666 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
W20240503 08:00:12.228256 12881 consensus_peers.cc:485] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcdde0a31f7d4fd0bf20c894f215e3b6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.235586 13208 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.237527 12815 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.237993 15762 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.241899 13143 consensus_peers.cc:485] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7bf40aa0614923a19feed8bab430fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.247164 12945 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.247670 15762 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.249298 13077 consensus_queue.cc:237] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.250804 12945 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.251008 12813 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.251271 14980 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.251443 15833 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.256132 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d61aa445a50417b82916019c2a4298c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.259186 12813 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.261142 13077 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.261430 15762 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.261618 15386 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.263248 15833 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.263594 12945 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.264303 12946 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.265308 12944 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.265782 15386 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.266503 12815 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.268353 12815 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.269847 12666 catalog_manager.cc:5484] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.277802 12666 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.285212 15843 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.288138 12945 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.288583 12666 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
W20240503 08:00:12.289551 12749 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8994c38d0f640e9bd5b5c322c603b9f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.293928 13077 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.329200 13208 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.332252 13077 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.333067 13076 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.334183 12945 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.335215 15689 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:12.335742 13141 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20cc823a5ce34916894cfde0a6614bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.337103 13208 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.337553 15689 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.337905 12945 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.338234 15833 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.339145 13143 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20cc823a5ce34916894cfde0a6614bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.339934 15833 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.340543 15689 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
W20240503 08:00:12.342106 13011 consensus_peers.cc:485] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e0d84a47cc34af9bdd6b955feb2c3ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.343287 12944 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.343741 12665 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.346503 13076 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.354280 15833 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.356544 12945 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.358139 13208 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.358858 12666 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.361233 12815 consensus_queue.cc:237] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.362126 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.362778 13076 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.362807 13208 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.363603 15386 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.363919 15386 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.369200 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:12.373003 12750 consensus_peers.cc:485] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a924487225dc4246aeb695060084096b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.378592 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.381373 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.381470 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:12.382994 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:12.385977 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.410652 15843 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.413158 13077 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.414949 13208 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.415347 12664 catalog_manager.cc:5484] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.431432 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:12.469082 12815 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.470880 12945 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.471493 13208 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.471582 15682 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.472180 15682 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.473829 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:12.477670 12750 consensus_peers.cc:485] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cee7c94dd0941d99d9dd6ceb39fc190. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.477944 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.484221 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.484261 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:12.487923 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:12.492161 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:12.505350 15682 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.508919 12945 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.510855 12669 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.510969 13208 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.517643 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:12.525255 12815 consensus_queue.cc:237] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.525686 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:00:12.527921 12750 consensus_peers.cc:485] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ab3fc1d0e6a42f38406bb75f69b8cbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.528538 12945 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.528919 15386 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.531329 13207 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.531673 15386 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.542253 13076 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.543845 12945 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.544732 12815 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.545431 15846 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.545817 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a60b868dcc504bb29d08e2752c8dd44d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.546860 15846 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.548673 15386 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.549929 13208 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.551358 12669 catalog_manager.cc:5484] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.554387 12945 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.562709 13208 consensus_queue.cc:237] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.564148 12945 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.564193 13076 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.564301 13143 consensus_peers.cc:485] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44b2c67014742af9a65a3576f509702. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.564509 15845 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.564687 14250 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.566787 15845 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.567727 15846 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.569147 13208 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.570060 12944 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.570449 12671 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.570653 13076 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.570658 13143 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c917c5849f54411eb55d69ab33456bf4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.570991 12815 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.573200 12945 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.574343 15927 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.575752 12945 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.576787 15927 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.577358 13077 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.578003 12669 catalog_manager.cc:5484] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.582307 15927 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.584080 13076 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.586211 12664 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.586750 12945 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.605834 12815 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.607448 12945 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.607484 13208 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.607888 15843 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.608599 15843 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.609513 12750 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5212dff7ded143a7923bb6176466b0fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.611191 12944 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.612813 12813 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.612820 12815 consensus_queue.cc:237] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.613381 13208 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.613976 15778 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.614617 12944 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:12.615072 12749 consensus_peers.cc:485] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ab3fc1d0e6a42f38406bb75f69b8cbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.615387 15843 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.615798 15843 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
W20240503 08:00:12.616077 12750 consensus_peers.cc:485] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ab3fc1d0e6a42f38406bb75f69b8cbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.616343 15778 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.617132 13208 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:12.617576 12879 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e4f2a993164fb6b1df674311698006. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.617642 15682 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.618422 12668 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.620559 13208 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.620965 15386 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.621241 12815 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.621415 13206 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.622521 12945 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.622553 15778 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.622907 13076 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.623438 15682 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.623615 12945 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
W20240503 08:00:12.624740 12750 consensus_peers.cc:485] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 389b40c2136d45ab86ccf49df8d1b71e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.624972 12664 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.627622 12664 catalog_manager.cc:5484] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.628858 13206 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.629025 15986 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.629082 12815 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.630792 13076 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.630820 12944 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.631536 15682 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
W20240503 08:00:12.632725 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a60b868dcc504bb29d08e2752c8dd44d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.632772 12815 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.633188 12944 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.633474 15846 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:12.633594 13011 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a60b868dcc504bb29d08e2752c8dd44d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.634099 13076 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.635428 15833 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.637137 12664 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.637447 12946 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.638248 15846 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.640899 12946 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.641418 12664 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.643630 12815 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.663605 13206 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.665127 13076 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.665361 16211 ts_tablet_manager.cc:902] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:12.665519 15845 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.666288 16211 tablet_copy_client.cc:308] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33937
I20240503 08:00:12.666649 12815 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.667052 15845 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.669425 13142 consensus_peers.cc:485] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02d3e4066cd14aa18619310a4490246b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.670687 13077 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.672382 12946 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.672454 12966 tablet_copy_service.cc:140] P 0cf63591cfc244028e1fc368c2bca079: Received BeginTabletCopySession request for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:50305)
I20240503 08:00:12.672586 12966 tablet_copy_service.cc:161] P 0cf63591cfc244028e1fc368c2bca079: Beginning new tablet copy session on tablet dcdde0a31f7d4fd0bf20c894f215e3b6 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:50305: session id = 82cd33f2bae4419b946095f8eb93e1e4-dcdde0a31f7d4fd0bf20c894f215e3b6
I20240503 08:00:12.672719 12815 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.673990 12966 tablet_copy_source_session.cc:205] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.674176 13206 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.674337 12749 consensus_peers.cc:485] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a2990625964c0e8b79780890b0e6c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.674548 15386 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.675007 13077 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.675251 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 076781af2ac14ddc87d31e3ec77db08f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.675351 15386 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.675787 13206 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.675885 14980 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.676267 14980 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.677498 12815 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.678834 15845 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.679005 12944 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.679358 15843 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.679585 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdde0a31f7d4fd0bf20c894f215e3b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:00:12.680672 12750 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5212dff7ded143a7923bb6176466b0fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.681159 13076 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
W20240503 08:00:12.681666 12749 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5212dff7ded143a7923bb6176466b0fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.683923 12664 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.685151 12815 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.685647 13208 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.687479 14980 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.687801 16153 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.689661 13206 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.690212 16211 tablet_copy_client.cc:783] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.690429 16211 tablet_copy_client.cc:647] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.691577 15386 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.692715 16211 tablet_copy_client.cc:523] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.693044 12669 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.694466 13077 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.694756 12668 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.696566 13208 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.697700 16211 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:12.697944 16248 ts_tablet_manager.cc:902] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:12.698307 12946 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.698724 16248 tablet_copy_client.cc:308] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:12.700333 16153 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.700843 13206 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.704331 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet d8994c38d0f640e9bd5b5c322c603b9f from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:12.704460 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet d8994c38d0f640e9bd5b5c322c603b9f from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-d8994c38d0f640e9bd5b5c322c603b9f
I20240503 08:00:12.704573 12946 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.706349 12835 tablet_copy_source_session.cc:205] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.707587 12671 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.714687 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.715570 13208 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.716233 16283 ts_tablet_manager.cc:902] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:12.716889 16283 tablet_copy_client.cc:308] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
W20240503 08:00:12.717270 13143 consensus_peers.cc:485] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c84f7a57e0945d9a25a70798f054bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.717958 13077 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.718333 14250 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.719938 12946 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.720614 14250 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.721526 16248 tablet_copy_client.cc:783] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.721709 16248 tablet_copy_client.cc:647] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.723665 16248 tablet_copy_client.cc:523] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.724014 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 20cc823a5ce34916894cfde0a6614bd1 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:39891)
I20240503 08:00:12.724140 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 20cc823a5ce34916894cfde0a6614bd1 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:39891: session id = eae9f53215a34bc88abe7b3c1c223319-20cc823a5ce34916894cfde0a6614bd1
I20240503 08:00:12.725481 13228 tablet_copy_source_session.cc:205] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.725643 16248 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:12.726497 16211 log.cc:824] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.727499 14250 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.729703 13077 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.732089 12671 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.732192 16248 log.cc:824] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.732259 12946 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.733593 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cc823a5ce34916894cfde0a6614bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.740223 16283 tablet_copy_client.cc:783] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.740653 16283 tablet_copy_client.cc:647] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.743690 16283 tablet_copy_client.cc:523] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.749037 16283 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:12.767962 16283 log.cc:824] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.774024 16211 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.774451 16211 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:12.774498 16248 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.774762 16211 ts_tablet_manager.cc:1367] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.077s user 0.030s sys 0.004s
I20240503 08:00:12.774830 16248 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:12.775162 16248 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.050s user 0.034s sys 0.000s
I20240503 08:00:12.776898 16211 raft_consensus.cc:348] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.777148 16211 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.777365 16248 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.777571 16248 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.777570 16211 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.777951 16248 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.778687 16248 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:00:12.779415 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-d8994c38d0f640e9bd5b5c322c603b9f received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:12.779505 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-d8994c38d0f640e9bd5b5c322c603b9f on tablet d8994c38d0f640e9bd5b5c322c603b9f with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:12.780138 16211 ts_tablet_manager.cc:1398] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:00:12.780750 12966 tablet_copy_service.cc:342] P 0cf63591cfc244028e1fc368c2bca079: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-dcdde0a31f7d4fd0bf20c894f215e3b6 received from {username='slave'} at 127.0.99.133:50305
I20240503 08:00:12.780838 12966 tablet_copy_service.cc:433] P 0cf63591cfc244028e1fc368c2bca079: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-dcdde0a31f7d4fd0bf20c894f215e3b6 on tablet dcdde0a31f7d4fd0bf20c894f215e3b6 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:12.784839 13077 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.786476 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.786686 16248 ts_tablet_manager.cc:902] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:12.786777 13208 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.787117 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a45748a98c3447d3aa2e8de94981a745. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.787312 16248 tablet_copy_client.cc:308] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:12.787287 16167 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.787578 16167 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.792321 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:55423)
I20240503 08:00:12.792461 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 0e0d84a47cc34af9bdd6b955feb2c3ba from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:55423: session id = 8398e6f89bf5419d81ed5bed991da0cf-0e0d84a47cc34af9bdd6b955feb2c3ba
I20240503 08:00:12.792903 12815 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.793856 13097 tablet_copy_source_session.cc:205] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:00:12.794616 12749 consensus_peers.cc:485] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a2990625964c0e8b79780890b0e6c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.795002 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:00:12.794627 12750 consensus_peers.cc:485] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a2990625964c0e8b79780890b0e6c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.795233 13077 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.795603 15386 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.796191 13208 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.796559 15386 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.797102 16248 tablet_copy_client.cc:783] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.797236 16248 tablet_copy_client.cc:647] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.799211 15386 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.801347 13077 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.802434 12664 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.802474 13208 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.805797 16167 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.806957 16248 tablet_copy_client.cc:523] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.807646 13206 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.808208 16248 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:12.808226 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.813592 12668 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:12.815264 13206 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.816584 12946 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.816591 13077 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.816969 15689 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.817225 15927 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.819000 13143 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70c5de226c7a4b349eef86a79d172851. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.824651 16211 ts_tablet_manager.cc:902] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:12.825037 16211 tablet_copy_client.cc:308] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:12.827368 16358 ts_tablet_manager.cc:902] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:12.827883 16358 tablet_copy_client.cc:308] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:12.829720 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet a924487225dc4246aeb695060084096b from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:12.829813 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet a924487225dc4246aeb695060084096b from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-a924487225dc4246aeb695060084096b
I20240503 08:00:12.830682 12835 tablet_copy_source_session.cc:205] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.831506 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a924487225dc4246aeb695060084096b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.831930 15762 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.832653 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 7d61aa445a50417b82916019c2a4298c from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:12.832759 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 7d61aa445a50417b82916019c2a4298c from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-7d61aa445a50417b82916019c2a4298c
I20240503 08:00:12.833621 16211 tablet_copy_client.cc:783] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.833648 13097 tablet_copy_source_session.cc:205] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.833762 16211 tablet_copy_client.cc:647] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.834460 13077 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.834599 16248 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.834705 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d61aa445a50417b82916019c2a4298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.834841 16248 tablet_bootstrap.cc:492] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:12.835031 16248 ts_tablet_manager.cc:1367] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.000s
I20240503 08:00:12.835635 16248 raft_consensus.cc:348] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.835753 16248 raft_consensus.cc:729] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.835778 12669 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.835884 16248 consensus_queue.cc:260] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.836146 16248 ts_tablet_manager.cc:1398] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:12.837383 16358 tablet_copy_client.cc:783] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.837565 16358 tablet_copy_client.cc:647] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.837774 12946 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.837867 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-0e0d84a47cc34af9bdd6b955feb2c3ba received from {username='slave'} at 127.0.99.134:55423
I20240503 08:00:12.837945 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-0e0d84a47cc34af9bdd6b955feb2c3ba on tablet 0e0d84a47cc34af9bdd6b955feb2c3ba with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:12.839310 16379 ts_tablet_manager.cc:902] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:12.841872 16379 tablet_copy_client.cc:308] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:12.843322 16283 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.843626 16283 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:12.843887 16283 ts_tablet_manager.cc:1367] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.095s user 0.031s sys 0.004s
I20240503 08:00:12.846065 16283 raft_consensus.cc:348] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.846313 16283 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.846401 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 9c7bf40aa0614923a19feed8bab430fa from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:59733)
I20240503 08:00:12.846490 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 9c7bf40aa0614923a19feed8bab430fa from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:59733: session id = b052e89b63c24258872c832a6a0d4d4d-9c7bf40aa0614923a19feed8bab430fa
I20240503 08:00:12.846735 16283 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.847303 13228 tablet_copy_source_session.cc:205] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.848023 16211 tablet_copy_client.cc:523] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.849350 16211 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:12.850555 16379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7bf40aa0614923a19feed8bab430fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.853013 16283 ts_tablet_manager.cc:1398] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:00:12.853832 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-20cc823a5ce34916894cfde0a6614bd1 received from {username='slave'} at 127.0.99.136:39891
I20240503 08:00:12.853924 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-20cc823a5ce34916894cfde0a6614bd1 on tablet 20cc823a5ce34916894cfde0a6614bd1 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:12.858839 16379 tablet_copy_client.cc:783] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.859082 16379 tablet_copy_client.cc:647] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.860383 16379 tablet_copy_client.cc:523] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.861940 16379 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:12.864081 12815 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.866567 13077 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.867393 16358 tablet_copy_client.cc:523] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.868788 16358 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:12.869455 16379 log.cc:824] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.871640 12946 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.872506 15386 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.873466 15843 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:12.873770 12750 consensus_peers.cc:485] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 389b40c2136d45ab86ccf49df8d1b71e. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:12.873867 12750 consensus_peers.cc:485] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 389b40c2136d45ab86ccf49df8d1b71e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.877226 15843 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.879801 13077 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.882364 12669 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.885880 12946 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:12.899138 16379 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.899560 16379 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:12.899925 16379 ts_tablet_manager.cc:1367] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.004s
I20240503 08:00:12.903017 16379 raft_consensus.cc:348] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.903471 16379 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.904042 16379 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.905129 16379 ts_tablet_manager.cc:1398] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:00:12.919625 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-9c7bf40aa0614923a19feed8bab430fa received from {username='slave'} at 127.0.99.135:59733
I20240503 08:00:12.919718 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-9c7bf40aa0614923a19feed8bab430fa on tablet 9c7bf40aa0614923a19feed8bab430fa with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:12.921756 16427 ts_tablet_manager.cc:902] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:12.922152 16427 tablet_copy_client.cc:308] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:12.927134 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:00:12.927237 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:12.928185 12835 tablet_copy_source_session.cc:205] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.929100 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.931864 16427 tablet_copy_client.cc:783] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.932083 16427 tablet_copy_client.cc:647] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.932910 13208 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.934502 12815 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:12.934772 13143 consensus_peers.cc:485] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c630e5cf7c0419cbf5d13413d22d317. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.935001 15689 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.935715 13077 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.936136 15689 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.936336 16211 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.936584 16211 tablet_bootstrap.cc:492] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:12.936736 16358 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.936761 16211 ts_tablet_manager.cc:1367] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.087s user 0.022s sys 0.004s
I20240503 08:00:12.937007 16358 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:12.937158 16358 ts_tablet_manager.cc:1367] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.068s user 0.023s sys 0.004s
I20240503 08:00:12.937412 16211 raft_consensus.cc:348] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.937530 16211 raft_consensus.cc:729] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.937659 16211 consensus_queue.cc:260] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:12.938853 16427 tablet_copy_client.cc:523] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.937922 16211 ts_tablet_manager.cc:1398] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.939399 16448 ts_tablet_manager.cc:902] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:12.939507 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-a924487225dc4246aeb695060084096b received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:12.939592 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-a924487225dc4246aeb695060084096b on tablet a924487225dc4246aeb695060084096b with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:12.939992 16448 tablet_copy_client.cc:308] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:12.940553 16427 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:12.941455 16358 raft_consensus.cc:348] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.941628 16358 raft_consensus.cc:729] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.941833 16358 consensus_queue.cc:260] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:12.945708 16358 ts_tablet_manager.cc:1398] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:00:12.945969 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 20cc823a5ce34916894cfde0a6614bd1 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53605)
I20240503 08:00:12.946064 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 20cc823a5ce34916894cfde0a6614bd1 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53605: session id = 82cd33f2bae4419b946095f8eb93e1e4-20cc823a5ce34916894cfde0a6614bd1
I20240503 08:00:12.946827 13228 tablet_copy_source_session.cc:205] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.947078 14250 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.947238 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-7d61aa445a50417b82916019c2a4298c received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:12.947300 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-7d61aa445a50417b82916019c2a4298c on tablet 7d61aa445a50417b82916019c2a4298c with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:12.947978 13077 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.949656 12815 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.954504 12668 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.955978 16448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cc823a5ce34916894cfde0a6614bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.958902 16448 tablet_copy_client.cc:783] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:12.959117 16448 tablet_copy_client.cc:647] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:12.960883 16448 tablet_copy_client.cc:523] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.962302 16448 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:12.970018 13076 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
W20240503 08:00:12.972859 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a45748a98c3447d3aa2e8de94981a745. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.972999 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.973387 16167 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.975433 13208 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:12.976698 13077 consensus_queue.cc:237] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:12.980279 12815 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.980465 14980 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:12.980856 16167 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:12.981319 13011 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672fc9981d074f7e8f2324ba0b06be57. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.982774 13208 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:12.983903 16167 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:12.984405 16167 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.985370 16427 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.985647 16427 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:12.985854 16427 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.000s
I20240503 08:00:12.986477 16427 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.986577 16427 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.986694 16427 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:12.986979 16427 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.987663 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-5cee7c94dd0941d99d9dd6ceb39fc190 received from {username='slave'} at 127.0.99.135:45337
I20240503 08:00:12.987731 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-5cee7c94dd0941d99d9dd6ceb39fc190 on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:12.988991 13207 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
W20240503 08:00:12.991518 13012 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a45748a98c3447d3aa2e8de94981a745. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:12.992154 14980 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.992828 13206 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.993954 12815 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.994307 16211 ts_tablet_manager.cc:902] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:12.994421 12664 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.994782 16211 tablet_copy_client.cc:308] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:12.995287 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:12.997642 12672 catalog_manager.cc:5484] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:12.998219 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet d44b2c67014742af9a65a3576f509702 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53605)
I20240503 08:00:12.998307 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet d44b2c67014742af9a65a3576f509702 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53605: session id = 82cd33f2bae4419b946095f8eb93e1e4-d44b2c67014742af9a65a3576f509702
I20240503 08:00:12.999133 13228 tablet_copy_source_session.cc:205] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:12.999933 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44b2c67014742af9a65a3576f509702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.002027 16211 tablet_copy_client.cc:783] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.002152 16211 tablet_copy_client.cc:647] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.004498 16211 tablet_copy_client.cc:523] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.005654 16211 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.035885 13077 consensus_queue.cc:237] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.037470 13206 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:13.037672 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672fc9981d074f7e8f2324ba0b06be57. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:13.037873 13011 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672fc9981d074f7e8f2324ba0b06be57. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.037889 16486 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.039747 16211 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.040069 16211 tablet_bootstrap.cc:492] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.040066 12815 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.040228 16211 ts_tablet_manager.cc:1367] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.035s user 0.030s sys 0.000s
I20240503 08:00:13.040670 16211 raft_consensus.cc:348] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.040781 16211 raft_consensus.cc:729] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.040901 16211 consensus_queue.cc:260] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.040956 15832 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.041254 16211 ts_tablet_manager.cc:1398] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:13.041832 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-d44b2c67014742af9a65a3576f509702 received from {username='slave'} at 127.0.99.133:53605
I20240503 08:00:13.041913 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-d44b2c67014742af9a65a3576f509702 on tablet d44b2c67014742af9a65a3576f509702 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.042858 15832 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.043154 12815 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.045580 12664 catalog_manager.cc:5484] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.046010 13206 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.051518 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:13.051558 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:13.051924 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:13.085826 16448 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.086143 16448 tablet_bootstrap.cc:492] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.091940 16448 ts_tablet_manager.cc:1367] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.130s user 0.024s sys 0.004s
I20240503 08:00:13.092494 16448 raft_consensus.cc:348] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.092907 16448 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.093062 16448 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.093874 16448 ts_tablet_manager.cc:1398] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.094344 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-20cc823a5ce34916894cfde0a6614bd1 received from {username='slave'} at 127.0.99.133:53605
I20240503 08:00:13.094417 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-20cc823a5ce34916894cfde0a6614bd1 on tablet 20cc823a5ce34916894cfde0a6614bd1 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.095966 16427 ts_tablet_manager.cc:902] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:13.096383 16427 tablet_copy_client.cc:308] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:13.096853 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 1c84f7a57e0945d9a25a70798f054bd1 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:59733)
I20240503 08:00:13.096944 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 1c84f7a57e0945d9a25a70798f054bd1 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:59733: session id = b052e89b63c24258872c832a6a0d4d4d-1c84f7a57e0945d9a25a70798f054bd1
I20240503 08:00:13.097775 13228 tablet_copy_source_session.cc:205] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.098651 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c84f7a57e0945d9a25a70798f054bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.100895 16427 tablet_copy_client.cc:783] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.101052 16427 tablet_copy_client.cc:647] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.102546 16427 tablet_copy_client.cc:523] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.103765 16427 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.111332 16358 ts_tablet_manager.cc:902] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.111861 16358 tablet_copy_client.cc:308] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.112469 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 076781af2ac14ddc87d31e3ec77db08f from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:13.112584 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 076781af2ac14ddc87d31e3ec77db08f from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-076781af2ac14ddc87d31e3ec77db08f
I20240503 08:00:13.113559 13097 tablet_copy_source_session.cc:205] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.114602 16283 ts_tablet_manager.cc:902] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.114732 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.115090 16283 tablet_copy_client.cc:308] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.116839 16358 tablet_copy_client.cc:783] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.116968 16358 tablet_copy_client.cc:647] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.117278 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a60b868dcc504bb29d08e2752c8dd44d from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:13.117373 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a60b868dcc504bb29d08e2752c8dd44d from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-a60b868dcc504bb29d08e2752c8dd44d
I20240503 08:00:13.118131 13097 tablet_copy_source_session.cc:205] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.119525 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.121250 16283 tablet_copy_client.cc:783] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.121371 16283 tablet_copy_client.cc:647] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.126829 16379 ts_tablet_manager.cc:902] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.127378 16379 tablet_copy_client.cc:308] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.128027 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:00:13.128125 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-6ab3fc1d0e6a42f38406bb75f69b8cbc
I20240503 08:00:13.128700 16448 ts_tablet_manager.cc:902] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:13.128980 12835 tablet_copy_source_session.cc:205] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.129101 16448 tablet_copy_client.cc:308] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:13.129369 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:13.129493 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet c917c5849f54411eb55d69ab33456bf4 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53605)
I20240503 08:00:13.129577 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet c917c5849f54411eb55d69ab33456bf4 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53605: session id = 82cd33f2bae4419b946095f8eb93e1e4-c917c5849f54411eb55d69ab33456bf4
I20240503 08:00:13.130000 16379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.130483 13228 tablet_copy_source_session.cc:205] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.131019 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.131340 16448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.131971 16379 tablet_copy_client.cc:783] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.132095 16379 tablet_copy_client.cc:647] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.133643 16448 tablet_copy_client.cc:783] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.133769 16448 tablet_copy_client.cc:647] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.134054 16358 tablet_copy_client.cc:523] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.135363 16358 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:13.138377 12813 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.139865 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:13.140022 12750 consensus_peers.cc:485] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cee7c94dd0941d99d9dd6ceb39fc190. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.140302 15386 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.141012 14789 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.141573 15386 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.143852 16248 ts_tablet_manager.cc:902] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.143926 16379 tablet_copy_client.cc:523] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.144271 16248 tablet_copy_client.cc:308] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.144469 13206 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.144868 15986 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.145066 16379 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.145077 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a60b868dcc504bb29d08e2752c8dd44d from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:55423)
I20240503 08:00:13.145172 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a60b868dcc504bb29d08e2752c8dd44d from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:55423: session id = 8398e6f89bf5419d81ed5bed991da0cf-a60b868dcc504bb29d08e2752c8dd44d
I20240503 08:00:13.146296 13097 tablet_copy_source_session.cc:205] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.146620 15986 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.147195 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.147363 13206 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.149057 16448 tablet_copy_client.cc:523] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.149183 16248 tablet_copy_client.cc:783] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.149307 16248 tablet_copy_client.cc:647] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.150223 16448 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.150334 14658 raft_consensus.cc:1203] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.153963 12946 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.158883 12666 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.160478 16605 ts_tablet_manager.cc:902] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:13.161017 16605 tablet_copy_client.cc:308] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:13.163415 14789 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.165839 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 02d3e4066cd14aa18619310a4490246b from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:42527)
I20240503 08:00:13.165935 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 02d3e4066cd14aa18619310a4490246b from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:42527: session id = 8398e6f89bf5419d81ed5bed991da0cf-02d3e4066cd14aa18619310a4490246b
I20240503 08:00:13.166738 13228 tablet_copy_source_session.cc:205] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.167687 16427 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.167923 16427 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.168071 16427 ts_tablet_manager.cc:1367] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.064s user 0.024s sys 0.000s
I20240503 08:00:13.169315 16427 raft_consensus.cc:348] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.169417 16427 raft_consensus.cc:729] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.169817 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3e4066cd14aa18619310a4490246b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.170289 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.186086 16248 tablet_copy_client.cc:523] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.187686 16248 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.191887 16283 tablet_copy_client.cc:523] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.193240 16283 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:13.193372 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.217592 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.218461 14527 raft_consensus.cc:1203] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.223731 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:13.227119 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:13.237380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:13.243948 16605 tablet_copy_client.cc:783] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.244144 16605 tablet_copy_client.cc:647] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.250131 16358 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.251082 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:13.252638 16211 ts_tablet_manager.cc:902] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:13.253152 16211 tablet_copy_client.cc:308] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:13.253576 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 70c5de226c7a4b349eef86a79d172851 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53605)
I20240503 08:00:13.253670 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 70c5de226c7a4b349eef86a79d172851 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53605: session id = 82cd33f2bae4419b946095f8eb93e1e4-70c5de226c7a4b349eef86a79d172851
I20240503 08:00:13.254673 13228 tablet_copy_source_session.cc:205] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.255587 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.255735 16427 consensus_queue.cc:260] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.256618 16427 ts_tablet_manager.cc:1398] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.088s user 0.002s sys 0.000s
I20240503 08:00:13.257078 13227 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-1c84f7a57e0945d9a25a70798f054bd1 received from {username='slave'} at 127.0.99.135:59733
I20240503 08:00:13.257158 13227 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-1c84f7a57e0945d9a25a70798f054bd1 on tablet 1c84f7a57e0945d9a25a70798f054bd1 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.258133 16211 tablet_copy_client.cc:783] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.258306 16211 tablet_copy_client.cc:647] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.260315 16379 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.260571 16379 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.260753 16379 ts_tablet_manager.cc:1367] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.116s user 0.026s sys 0.000s
I20240503 08:00:13.261746 16358 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:13.261940 16358 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.127s user 0.021s sys 0.005s
I20240503 08:00:13.262447 16358 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.262566 16358 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.262738 16358 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.263759 16605 tablet_copy_client.cc:523] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.265259 16605 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.266322 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.270274 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:13.276073 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.282271 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.290236 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.295212 16645 ts_tablet_manager.cc:902] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.250416 16646 ts_tablet_manager.cc:902] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:13.296280 16645 tablet_copy_client.cc:308] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.296284 16646 tablet_copy_client.cc:308] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33937
I20240503 08:00:13.296789 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:13.296886 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-6ab3fc1d0e6a42f38406bb75f69b8cbc
I20240503 08:00:13.297760 12835 tablet_copy_source_session.cc:205] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.252308 14920 raft_consensus.cc:1203] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.301201 14527 raft_consensus.cc:1203] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.301697 16427 ts_tablet_manager.cc:902] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.302102 16427 tablet_copy_client.cc:308] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.303452 12966 tablet_copy_service.cc:140] P 0cf63591cfc244028e1fc368c2bca079: Received BeginTabletCopySession request for tablet 48e4f2a993164fb6b1df674311698006 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:37555)
I20240503 08:00:13.303525 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5212dff7ded143a7923bb6176466b0fa from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:00:13.303560 12966 tablet_copy_service.cc:161] P 0cf63591cfc244028e1fc368c2bca079: Beginning new tablet copy session on tablet 48e4f2a993164fb6b1df674311698006 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:37555: session id = eae9f53215a34bc88abe7b3c1c223319-48e4f2a993164fb6b1df674311698006
I20240503 08:00:13.303607 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5212dff7ded143a7923bb6176466b0fa from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-5212dff7ded143a7923bb6176466b0fa
I20240503 08:00:13.304466 12966 tablet_copy_source_session.cc:205] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.304466 12835 tablet_copy_source_session.cc:205] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.305402 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.305402 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.306210 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.308100 16427 tablet_copy_client.cc:783] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.308180 16646 tablet_copy_client.cc:783] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.308270 16427 tablet_copy_client.cc:647] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.308418 16646 tablet_copy_client.cc:647] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.315055 16358 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.053s user 0.001s sys 0.000s
I20240503 08:00:13.315567 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-076781af2ac14ddc87d31e3ec77db08f received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:13.315650 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-076781af2ac14ddc87d31e3ec77db08f on tablet 076781af2ac14ddc87d31e3ec77db08f with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:13.320125 16645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.321745 14527 raft_consensus.cc:1203] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.322795 16645 tablet_copy_client.cc:783] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.324431 16211 tablet_copy_client.cc:523] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.325857 16211 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.327368 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.329883 16646 tablet_copy_client.cc:523] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.331423 16646 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:13.261178 16379 raft_consensus.cc:348] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.331840 16379 raft_consensus.cc:729] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.332046 16379 consensus_queue.cc:260] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.339581 16645 tablet_copy_client.cc:647] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.341329 16645 tablet_copy_client.cc:523] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.341919 16379 ts_tablet_manager.cc:1398] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.081s user 0.001s sys 0.000s
I20240503 08:00:13.342392 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-6ab3fc1d0e6a42f38406bb75f69b8cbc received from {username='slave'} at 127.0.99.135:45337
I20240503 08:00:13.342486 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-6ab3fc1d0e6a42f38406bb75f69b8cbc on tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.346093 16645 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.353554 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.356340 16448 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.356603 16448 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.356761 16448 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.207s user 0.022s sys 0.003s
I20240503 08:00:13.357175 16448 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.357298 16448 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.357690 16448 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.365833 16448 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:13.368247 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:13.369544 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-c917c5849f54411eb55d69ab33456bf4 received from {username='slave'} at 127.0.99.133:53605
I20240503 08:00:13.369621 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-c917c5849f54411eb55d69ab33456bf4 on tablet c917c5849f54411eb55d69ab33456bf4 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.370553 16448 ts_tablet_manager.cc:902] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.370972 16448 tablet_copy_client.cc:308] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.371425 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet d9a2990625964c0e8b79780890b0e6c3 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:13.371547 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet d9a2990625964c0e8b79780890b0e6c3 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-d9a2990625964c0e8b79780890b0e6c3
I20240503 08:00:13.372445 12835 tablet_copy_source_session.cc:205] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.373507 16448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a2990625964c0e8b79780890b0e6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.375787 16448 tablet_copy_client.cc:783] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.375928 16448 tablet_copy_client.cc:647] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.379098 16678 ts_tablet_manager.cc:902] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.379688 16656 ts_tablet_manager.cc:902] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.379791 16678 tablet_copy_client.cc:308] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.380199 16656 tablet_copy_client.cc:308] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.380368 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5212dff7ded143a7923bb6176466b0fa from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:13.380474 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5212dff7ded143a7923bb6176466b0fa from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-5212dff7ded143a7923bb6176466b0fa
I20240503 08:00:13.381199 16448 tablet_copy_client.cc:523] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.381423 12835 tablet_copy_source_session.cc:205] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.382306 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet d9a2990625964c0e8b79780890b0e6c3 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:13.382395 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet d9a2990625964c0e8b79780890b0e6c3 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-d9a2990625964c0e8b79780890b0e6c3
I20240503 08:00:13.382453 16448 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.383123 12835 tablet_copy_source_session.cc:205] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.384552 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.385152 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.385795 14658 raft_consensus.cc:1203] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.388672 16679 ts_tablet_manager.cc:902] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.389219 16679 tablet_copy_client.cc:308] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.389730 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 389b40c2136d45ab86ccf49df8d1b71e from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:13.389812 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 389b40c2136d45ab86ccf49df8d1b71e from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:13.390522 12835 tablet_copy_source_session.cc:205] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.391239 16656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a2990625964c0e8b79780890b0e6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.391428 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.393369 16656 tablet_copy_client.cc:783] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.393564 16656 tablet_copy_client.cc:647] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.393733 16679 tablet_copy_client.cc:783] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.394981 16679 tablet_copy_client.cc:647] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.395975 16678 tablet_copy_client.cc:783] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.384570 16427 tablet_copy_client.cc:523] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.400219 16427 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.400398 16678 tablet_copy_client.cc:647] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.401523 16678 tablet_copy_client.cc:523] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.402994 16679 tablet_copy_client.cc:523] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.403262 16678 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.405207 16679 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.408259 16656 tablet_copy_client.cc:523] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.412005 16656 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.415763 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:13.417407 16283 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.417663 16283 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:13.417845 16283 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.225s user 0.026s sys 0.000s
I20240503 08:00:13.418187 16379 ts_tablet_manager.cc:902] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:13.418296 16283 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.418422 16283 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.418610 16379 tablet_copy_client.cc:308] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:13.419067 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:59733)
I20240503 08:00:13.419174 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:59733: session id = b052e89b63c24258872c832a6a0d4d4d-7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:13.419983 13228 tablet_copy_source_session.cc:205] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.421087 16379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.423242 16379 tablet_copy_client.cc:783] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.423375 16379 tablet_copy_client.cc:647] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.424669 16283 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.425653 16211 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.426494 16283 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:13.426704 14920 raft_consensus.cc:1203] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.435422 16211 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.435618 16211 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.110s user 0.026s sys 0.000s
I20240503 08:00:13.436549 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-a60b868dcc504bb29d08e2752c8dd44d received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:13.436628 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-a60b868dcc504bb29d08e2752c8dd44d on tablet a60b868dcc504bb29d08e2752c8dd44d with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:13.436144 16211 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.438215 16211 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.438355 16211 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.438637 16211 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:13.438676 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:13.440714 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-70c5de226c7a4b349eef86a79d172851 received from {username='slave'} at 127.0.99.133:53605
I20240503 08:00:13.440780 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-70c5de226c7a4b349eef86a79d172851 on tablet 70c5de226c7a4b349eef86a79d172851 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.447875 14789 raft_consensus.cc:1203] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.451071 16283 ts_tablet_manager.cc:902] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.451450 16283 tablet_copy_client.cc:308] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.451838 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 672fc9981d074f7e8f2324ba0b06be57 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:13.451939 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 672fc9981d074f7e8f2324ba0b06be57 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-672fc9981d074f7e8f2324ba0b06be57
I20240503 08:00:13.452793 13097 tablet_copy_source_session.cc:205] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.453636 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672fc9981d074f7e8f2324ba0b06be57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.455900 16283 tablet_copy_client.cc:783] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.456040 16283 tablet_copy_client.cc:647] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.464370 16605 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.467149 14527 raft_consensus.cc:1203] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.472219 16715 ts_tablet_manager.cc:902] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.472885 16715 tablet_copy_client.cc:308] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.473445 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 389b40c2136d45ab86ccf49df8d1b71e from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:00:13.473538 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 389b40c2136d45ab86ccf49df8d1b71e from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:13.474393 12835 tablet_copy_source_session.cc:205] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.475345 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.476562 16283 tablet_copy_client.cc:523] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.478070 16715 tablet_copy_client.cc:783] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.480067 16283 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:13.486069 16605 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.486369 16605 ts_tablet_manager.cc:1367] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.221s user 0.026s sys 0.000s
I20240503 08:00:13.486830 16605 raft_consensus.cc:348] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.487035 16605 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.487237 16605 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.488121 16605 ts_tablet_manager.cc:1398] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.488992 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-02d3e4066cd14aa18619310a4490246b received from {username='slave'} at 127.0.99.134:42527
I20240503 08:00:13.489075 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-02d3e4066cd14aa18619310a4490246b on tablet 02d3e4066cd14aa18619310a4490246b with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.492671 16448 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.492947 16448 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.493111 16448 ts_tablet_manager.cc:1367] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.111s user 0.023s sys 0.004s
I20240503 08:00:13.493562 16448 raft_consensus.cc:348] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.493687 16448 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.478837 16379 tablet_copy_client.cc:523] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.496052 16448 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.496345 16448 ts_tablet_manager.cc:1398] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:13.496541 16379 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.499138 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-d9a2990625964c0e8b79780890b0e6c3 received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:13.499222 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-d9a2990625964c0e8b79780890b0e6c3 on tablet d9a2990625964c0e8b79780890b0e6c3 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.479156 16714 ts_tablet_manager.cc:902] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.500851 16714 tablet_copy_client.cc:308] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.501986 16715 tablet_copy_client.cc:647] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.505776 16715 tablet_copy_client.cc:523] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.507169 16715 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.509254 16646 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.509557 16646 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:13.509732 16646 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.178s user 0.024s sys 0.000s
I20240503 08:00:13.510178 16646 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.510289 16646 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.510591 16646 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.464895 16248 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.478009 16717 ts_tablet_manager.cc:902] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.510888 16646 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:13.512185 16717 tablet_copy_client.cc:308] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.512670 12966 tablet_copy_service.cc:342] P 0cf63591cfc244028e1fc368c2bca079: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-48e4f2a993164fb6b1df674311698006 received from {username='slave'} at 127.0.99.136:37555
I20240503 08:00:13.512701 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 672fc9981d074f7e8f2324ba0b06be57 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:55423)
I20240503 08:00:13.512749 12966 tablet_copy_service.cc:433] P 0cf63591cfc244028e1fc368c2bca079: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-48e4f2a993164fb6b1df674311698006 on tablet 48e4f2a993164fb6b1df674311698006 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:13.512790 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 672fc9981d074f7e8f2324ba0b06be57 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:55423: session id = 8398e6f89bf5419d81ed5bed991da0cf-672fc9981d074f7e8f2324ba0b06be57
I20240503 08:00:13.513700 13097 tablet_copy_source_session.cc:205] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.515452 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672fc9981d074f7e8f2324ba0b06be57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.517942 16717 tablet_copy_client.cc:783] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.518143 16717 tablet_copy_client.cc:647] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.518950 16248 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.519239 16248 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.332s user 0.024s sys 0.004s
I20240503 08:00:13.519723 16248 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.519898 16248 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.520073 16248 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.520447 12946 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.521024 16248 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.521473 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-a60b868dcc504bb29d08e2752c8dd44d received from {username='slave'} at 127.0.99.134:55423
I20240503 08:00:13.521548 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-a60b868dcc504bb29d08e2752c8dd44d on tablet a60b868dcc504bb29d08e2752c8dd44d with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.523092 13077 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.523720 14442 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.525344 12815 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.525784 16679 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.526077 16679 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.526259 16679 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.121s user 0.016s sys 0.010s
I20240503 08:00:13.526321 14920 raft_consensus.cc:1203] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.526670 16679 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.526772 16679 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.526885 16679 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.527189 16679 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:13.527722 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-389b40c2136d45ab86ccf49df8d1b71e received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:13.527798 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-389b40c2136d45ab86ccf49df8d1b71e on tablet 389b40c2136d45ab86ccf49df8d1b71e with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.529414 14442 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.527892 14789 raft_consensus.cc:1203] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.531718 13206 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
W20240503 08:00:13.534412 13142 consensus_peers.cc:485] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c84f7a57e0945d9a25a70798f054bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.534493 16678 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.534566 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a45748a98c3447d3aa2e8de94981a745 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:44613)
I20240503 08:00:13.534657 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a45748a98c3447d3aa2e8de94981a745 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:44613: session id = b052e89b63c24258872c832a6a0d4d4d-a45748a98c3447d3aa2e8de94981a745
I20240503 08:00:13.535476 14789 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.535560 13097 tablet_copy_source_session.cc:205] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.535588 13077 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.537166 16717 tablet_copy_client.cc:523] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.538560 16717 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:13.538069 15845 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:00:13.541321 12946 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.541712 15845 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.543202 15689 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.545188 15845 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.547895 12665 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.550004 13077 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.553048 12946 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.556316 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.557090 16427 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.557343 16427 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.557513 16427 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.157s user 0.024s sys 0.000s
I20240503 08:00:13.557921 16427 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.558029 16427 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.558157 16427 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.558434 16427 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:13.559340 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-5212dff7ded143a7923bb6176466b0fa received from {username='slave'} at 127.0.99.135:45337
I20240503 08:00:13.559434 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-5212dff7ded143a7923bb6176466b0fa on tablet 5212dff7ded143a7923bb6176466b0fa with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.559902 16714 tablet_copy_client.cc:783] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.560083 16714 tablet_copy_client.cc:647] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.534833 16678 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.562322 16678 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.159s user 0.020s sys 0.006s
I20240503 08:00:13.562705 16714 tablet_copy_client.cc:523] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.563884 16714 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:13.566092 16678 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.567309 13076 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.568830 14442 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.566246 16678 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.569957 12665 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.570189 16678 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.571276 12815 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.571739 14920 raft_consensus.cc:1203] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.572613 16715 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.572892 16715 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.573041 16715 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.066s user 0.026s sys 0.000s
I20240503 08:00:13.573434 16715 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.574240 16678 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 08:00:13.575284 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5212dff7ded143a7923bb6176466b0fa received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:13.575358 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5212dff7ded143a7923bb6176466b0fa on tablet 5212dff7ded143a7923bb6176466b0fa with peer 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.578315 12880 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e7f229898c24572815177628720521f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.580006 16715 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.580197 16715 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.581036 16715 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:13.582209 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-389b40c2136d45ab86ccf49df8d1b71e received from {username='slave'} at 127.0.99.135:45337
I20240503 08:00:13.582286 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-389b40c2136d45ab86ccf49df8d1b71e on tablet 389b40c2136d45ab86ccf49df8d1b71e with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.583055 14789 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:13.588295 16646 ts_tablet_manager.cc:902] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:13.588725 16646 tablet_copy_client.cc:308] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:13.589236 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a45748a98c3447d3aa2e8de94981a745 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:13.589325 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a45748a98c3447d3aa2e8de94981a745 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-a45748a98c3447d3aa2e8de94981a745
I20240503 08:00:13.590227 13097 tablet_copy_source_session.cc:205] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.590574 16679 ts_tablet_manager.cc:902] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:13.591050 16679 tablet_copy_client.cc:308] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:13.591464 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:13.591545 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:13.592305 12835 tablet_copy_source_session.cc:205] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:13.592995 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.594048 14527 raft_consensus.cc:1203] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.594868 16646 tablet_copy_client.cc:783] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.595633 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.597261 16679 tablet_copy_client.cc:783] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:13.597404 16679 tablet_copy_client.cc:647] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.597854 16646 tablet_copy_client.cc:647] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:13.598702 16679 tablet_copy_client.cc:523] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.599862 16679 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:13.602460 16646 tablet_copy_client.cc:523] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:13.603868 16646 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:13.605777 16645 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.606191 16645 tablet_bootstrap.cc:492] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.606951 16645 ts_tablet_manager.cc:1367] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.261s user 0.028s sys 0.000s
I20240503 08:00:13.607430 16645 raft_consensus.cc:348] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.613168 16645 raft_consensus.cc:729] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.613390 16645 consensus_queue.cc:260] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:13.614432 16645 ts_tablet_manager.cc:1398] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:13.614867 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-6ab3fc1d0e6a42f38406bb75f69b8cbc received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:13.615001 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-6ab3fc1d0e6a42f38406bb75f69b8cbc on tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.622023 16717 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.622292 16717 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.622440 16717 ts_tablet_manager.cc:1367] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.084s user 0.024s sys 0.000s
I20240503 08:00:13.622795 16717 raft_consensus.cc:348] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.622905 16717 raft_consensus.cc:729] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.624380 16717 consensus_queue.cc:260] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.626988 16717 ts_tablet_manager.cc:1398] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:00:13.627418 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-672fc9981d074f7e8f2324ba0b06be57 received from {username='slave'} at 127.0.99.134:55423
I20240503 08:00:13.627511 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-672fc9981d074f7e8f2324ba0b06be57 on tablet 672fc9981d074f7e8f2324ba0b06be57 with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.629765 16656 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.636751 16379 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.637002 16379 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.637104 16656 tablet_bootstrap.cc:492] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:13.637171 16379 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.141s user 0.016s sys 0.006s
I20240503 08:00:13.637267 16656 ts_tablet_manager.cc:1367] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.225s user 0.025s sys 0.000s
I20240503 08:00:13.637612 16379 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.637619 16656 raft_consensus.cc:348] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.637725 16656 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.637725 16379 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.637858 16379 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.637890 16656 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.638165 16656 ts_tablet_manager.cc:1398] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.638556 16379 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:13.638983 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-d9a2990625964c0e8b79780890b0e6c3 received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:13.639065 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-d9a2990625964c0e8b79780890b0e6c3 on tablet d9a2990625964c0e8b79780890b0e6c3 with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.640421 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-7c630e5cf7c0419cbf5d13413d22d317 received from {username='slave'} at 127.0.99.135:59733
I20240503 08:00:13.640499 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-7c630e5cf7c0419cbf5d13413d22d317 on tablet 7c630e5cf7c0419cbf5d13413d22d317 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.647550 16283 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.647809 16283 tablet_bootstrap.cc:492] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:13.647986 16283 ts_tablet_manager.cc:1367] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.168s user 0.021s sys 0.004s
I20240503 08:00:13.648429 16283 raft_consensus.cc:348] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.648629 16283 raft_consensus.cc:729] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.648793 16283 consensus_queue.cc:260] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.649161 16283 ts_tablet_manager.cc:1398] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:13.649317 12815 consensus_queue.cc:237] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.649590 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-672fc9981d074f7e8f2324ba0b06be57 received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:13.649664 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-672fc9981d074f7e8f2324ba0b06be57 on tablet 672fc9981d074f7e8f2324ba0b06be57 with peer eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.651582 12750 consensus_peers.cc:485] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5480e544dca3428ab4fcf190577d5331. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.652453 13076 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.652786 13206 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.653264 15843 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.653692 15986 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.656466 14789 raft_consensus.cc:1203] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.660629 13077 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.661008 15986 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.662678 14658 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:13.662770 14920 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:13.663345 15833 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:13.663622 13206 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.664482 12815 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:13.665215 12672 catalog_manager.cc:5484] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.667205 13077 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.667552 15833 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:13.667830 15833 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:13.668736 12944 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:13.670725 15833 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:13.673883 16847 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.674327 12815 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.676131 14658 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.677444 16714 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.677721 16714 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:13.677874 16714 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.114s user 0.020s sys 0.005s
I20240503 08:00:13.677739 12665 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.678314 16714 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.678423 16714 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.679992 12944 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.684262 16714 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.684669 16714 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:00:13.685470 16646 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:13.685137 14920 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:13.685779 16646 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:13.685935 16646 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.082s user 0.025s sys 0.000s
I20240503 08:00:13.686702 16646 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.686802 16646 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.686976 16646 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.687268 16646 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.687816 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-a45748a98c3447d3aa2e8de94981a745 received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:13.687894 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-a45748a98c3447d3aa2e8de94981a745 on tablet a45748a98c3447d3aa2e8de94981a745 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:13.688887 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-a45748a98c3447d3aa2e8de94981a745 received from {username='slave'} at 127.0.99.135:44613
I20240503 08:00:13.688949 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-a45748a98c3447d3aa2e8de94981a745 on tablet a45748a98c3447d3aa2e8de94981a745 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.692462 13077 consensus_queue.cc:237] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.693074 16679 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 08:00:13.693243 13012 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a60b868dcc504bb29d08e2752c8dd44d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.693329 16679 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:13.693502 16679 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.094s user 0.023s sys 0.001s
I20240503 08:00:13.693908 16679 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.693954 12944 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.694027 16679 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.694181 16679 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:13.694371 12815 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.694416 16847 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.694705 16847 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:13.695287 13012 consensus_peers.cc:485] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b094d04434d24923806b0e9e209c3fca. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.697329 16679 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:00:13.698019 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5cee7c94dd0941d99d9dd6ceb39fc190 received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:13.698087 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5cee7c94dd0941d99d9dd6ceb39fc190 on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.700709 14658 raft_consensus.cc:1203] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.703341 16848 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.704572 14527 raft_consensus.cc:1203] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.705404 12944 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.707398 12815 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.708175 12672 catalog_manager.cc:5484] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.709183 13206 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.711848 12815 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:13.712167 13141 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 047aba81fad543ce8b562a184746b163. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.712251 16766 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.716733 12944 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.717058 14658 raft_consensus.cc:1203] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.718014 16766 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.726074 16766 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.726399 12815 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.728047 12944 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.729451 12667 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.734572 13206 consensus_queue.cc:237] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.736181 12815 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.736552 15689 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.737958 12944 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:13.738291 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d376ac1ea05c482ca8f03ddf0f5fcd47. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.738471 15689 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.747706 14920 raft_consensus.cc:1203] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.750591 15689 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.750891 12815 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.752450 12944 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.753304 12667 catalog_manager.cc:5484] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index -1 to 4, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.764739 13206 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:13.766515 14527 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.766639 13076 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.767165 16772 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.767277 12944 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.767618 16772 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.767853 16772 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:13.770608 13141 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c917c5849f54411eb55d69ab33456bf4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.771005 16766 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.771471 13076 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.774271 12670 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.775916 12944 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.783186 12815 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.785161 13206 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:13.785177 12750 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ac9fbe5dfd4a4e89912d7e947eca45. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.785578 15986 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.787395 13076 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.787770 15386 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.795773 14527 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:13.802970 15386 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.806663 12667 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.806978 13206 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.807025 13076 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.816506 13076 consensus_queue.cc:237] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.818048 12944 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:13.818084 14658 raft_consensus.cc:1203] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.818555 16846 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.818780 12815 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:13.819298 13012 consensus_peers.cc:485] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1151731adc9407b86d143d6d41f69b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.820552 16847 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:13.822496 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:13.825445 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:13.839262 16847 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.840873 14658 raft_consensus.cc:1203] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.841249 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.842108 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:13.842478 12815 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.845940 12944 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.846365 12666 catalog_manager.cc:5484] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index -1 to 4, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.850870 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.852471 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.853739 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.860942 14658 raft_consensus.cc:1203] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.867785 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:13.897087 14789 raft_consensus.cc:1203] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.897699 13076 consensus_queue.cc:237] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:13.899778 12944 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.900394 14920 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.900867 12813 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:13.901532 15833 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.902249 15833 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:13.902485 16486 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:13.905421 13012 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d61aa445a50417b82916019c2a4298c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:13.914268 14789 raft_consensus.cc:1203] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:13.915654 14527 raft_consensus.cc:1203] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.932186 16486 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.935474 12667 catalog_manager.cc:5484] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.936563 12944 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.938143 12813 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.939752 14920 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:13.954115 14920 raft_consensus.cc:1203] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.964290 14527 raft_consensus.cc:1203] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.976794 14789 raft_consensus.cc:1203] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:13.987691 16656 ts_tablet_manager.cc:902] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:13.988831 16656 tablet_copy_client.cc:308] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33937
I20240503 08:00:14.008774 12813 consensus_queue.cc:237] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.010622 13076 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:14.011335 13206 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:14.011890 12749 consensus_peers.cc:485] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5480e544dca3428ab4fcf190577d5331. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:14.012086 12750 consensus_peers.cc:485] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5480e544dca3428ab4fcf190577d5331. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:14.012411 15843 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:14.012614 15986 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:14.016541 15986 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.018334 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.019213 13076 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.019814 12667 catalog_manager.cc:5484] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.021512 12966 tablet_copy_service.cc:140] P 0cf63591cfc244028e1fc368c2bca079: Received BeginTabletCopySession request for tablet 9e7f229898c24572815177628720521f from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:58221)
I20240503 08:00:14.021616 12966 tablet_copy_service.cc:161] P 0cf63591cfc244028e1fc368c2bca079: Beginning new tablet copy session on tablet 9e7f229898c24572815177628720521f from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:58221: session id = 8398e6f89bf5419d81ed5bed991da0cf-9e7f229898c24572815177628720521f
I20240503 08:00:14.022544 12966 tablet_copy_source_session.cc:205] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.023993 16656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.024780 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.026261 16656 tablet_copy_client.cc:783] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.026414 16656 tablet_copy_client.cc:647] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.029697 16717 ts_tablet_manager.cc:902] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:14.030200 16717 tablet_copy_client.cc:308] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:14.030887 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 1c84f7a57e0945d9a25a70798f054bd1 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:42527)
I20240503 08:00:14.030993 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 1c84f7a57e0945d9a25a70798f054bd1 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:42527: session id = 8398e6f89bf5419d81ed5bed991da0cf-1c84f7a57e0945d9a25a70798f054bd1
I20240503 08:00:14.031000 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.031463 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.031755 13228 tablet_copy_source_session.cc:205] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.032066 13206 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.032896 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c84f7a57e0945d9a25a70798f054bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.034844 16717 tablet_copy_client.cc:783] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.036533 16717 tablet_copy_client.cc:647] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.040524 16656 tablet_copy_client.cc:523] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.040585 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.041738 16656 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:14.047981 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:14.056074 16717 tablet_copy_client.cc:523] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.056139 16679 ts_tablet_manager.cc:902] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:14.056548 16679 tablet_copy_client.cc:308] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:14.057452 16717 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:14.062464 14789 raft_consensus.cc:1203] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.087100 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.118337 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.119626 16714 ts_tablet_manager.cc:902] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:14.120060 16714 tablet_copy_client.cc:308] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:14.120822 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.120822 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet b094d04434d24923806b0e9e209c3fca from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:44613)
I20240503 08:00:14.120908 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet b094d04434d24923806b0e9e209c3fca from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:44613: session id = b052e89b63c24258872c832a6a0d4d4d-b094d04434d24923806b0e9e209c3fca
I20240503 08:00:14.121794 13097 tablet_copy_source_session.cc:205] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.123152 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b094d04434d24923806b0e9e209c3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.125308 16714 tablet_copy_client.cc:783] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.125444 16714 tablet_copy_client.cc:647] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.127445 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.134390 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.136250 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.138511 14527 raft_consensus.cc:1203] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.141377 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:14.141896 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.160195 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.180570 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a60b868dcc504bb29d08e2752c8dd44d from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:36949)
I20240503 08:00:14.180673 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a60b868dcc504bb29d08e2752c8dd44d from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:36949: session id = 82cd33f2bae4419b946095f8eb93e1e4-a60b868dcc504bb29d08e2752c8dd44d
I20240503 08:00:14.181555 13097 tablet_copy_source_session.cc:205] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.182711 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60b868dcc504bb29d08e2752c8dd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.183774 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.188853 16679 tablet_copy_client.cc:783] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.189000 16679 tablet_copy_client.cc:647] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.192317 16679 tablet_copy_client.cc:523] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.193089 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.193580 16679 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:14.196579 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.204754 14920 raft_consensus.cc:1203] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.206321 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.212368 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:14.212698 16714 tablet_copy_client.cc:523] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.214143 16714 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:14.224015 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.230871 16656 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.233353 16656 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:14.236515 16656 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.195s user 0.021s sys 0.006s
I20240503 08:00:14.236874 16717 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.237088 16717 tablet_bootstrap.cc:492] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:14.237057 16656 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.237162 16656 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.237221 16717 ts_tablet_manager.cc:1367] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.180s user 0.025s sys 0.000s
I20240503 08:00:14.237272 16656 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.237603 16717 raft_consensus.cc:348] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.237692 16717 raft_consensus.cc:729] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.238034 16656 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.238471 12966 tablet_copy_service.cc:342] P 0cf63591cfc244028e1fc368c2bca079: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-9e7f229898c24572815177628720521f received from {username='slave'} at 127.0.99.134:58221
I20240503 08:00:14.238554 12966 tablet_copy_service.cc:433] P 0cf63591cfc244028e1fc368c2bca079: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-9e7f229898c24572815177628720521f on tablet 9e7f229898c24572815177628720521f with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:14.239498 16717 consensus_queue.cc:260] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.241088 16717 ts_tablet_manager.cc:1398] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:14.241513 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-1c84f7a57e0945d9a25a70798f054bd1 received from {username='slave'} at 127.0.99.134:42527
I20240503 08:00:14.241585 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-1c84f7a57e0945d9a25a70798f054bd1 on tablet 1c84f7a57e0945d9a25a70798f054bd1 with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:14.242087 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.244325 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.257627 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.279067 17212 ts_tablet_manager.cc:902] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:14.279613 17212 tablet_copy_client.cc:308] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:14.280092 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet c917c5849f54411eb55d69ab33456bf4 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:39891)
I20240503 08:00:14.280185 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet c917c5849f54411eb55d69ab33456bf4 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:39891: session id = eae9f53215a34bc88abe7b3c1c223319-c917c5849f54411eb55d69ab33456bf4
I20240503 08:00:14.281009 13228 tablet_copy_source_session.cc:205] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.281955 17212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.284557 17212 tablet_copy_client.cc:783] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.284744 17212 tablet_copy_client.cc:647] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.302031 17214 ts_tablet_manager.cc:902] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:14.302610 17214 tablet_copy_client.cc:308] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:14.303067 17212 tablet_copy_client.cc:523] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.303148 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:00:14.303225 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-72ac9fbe5dfd4a4e89912d7e947eca45
I20240503 08:00:14.304087 12835 tablet_copy_source_session.cc:205] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.304574 17212 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:14.305898 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.309617 17214 tablet_copy_client.cc:783] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.309803 17214 tablet_copy_client.cc:647] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.324600 17214 tablet_copy_client.cc:523] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.326694 17214 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:14.332690 16714 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.333045 16714 tablet_bootstrap.cc:492] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:14.333240 16714 ts_tablet_manager.cc:1367] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.119s user 0.022s sys 0.003s
I20240503 08:00:14.334311 17243 ts_tablet_manager.cc:902] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:14.334882 17243 tablet_copy_client.cc:308] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:14.335382 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet 047aba81fad543ce8b562a184746b163 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:39891)
I20240503 08:00:14.335491 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet 047aba81fad543ce8b562a184746b163 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:39891: session id = eae9f53215a34bc88abe7b3c1c223319-047aba81fad543ce8b562a184746b163
I20240503 08:00:14.336330 13228 tablet_copy_source_session.cc:205] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.338295 17212 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.338552 17212 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:14.338725 17212 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.000s
I20240503 08:00:14.338997 17243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047aba81fad543ce8b562a184746b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.339243 17212 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.339363 17212 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.339520 17212 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.339808 17212 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:14.340404 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-c917c5849f54411eb55d69ab33456bf4 received from {username='slave'} at 127.0.99.136:39891
I20240503 08:00:14.340481 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-c917c5849f54411eb55d69ab33456bf4 on tablet c917c5849f54411eb55d69ab33456bf4 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:14.349892 17212 ts_tablet_manager.cc:902] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:14.350343 17212 tablet_copy_client.cc:308] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:14.350679 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:39891)
I20240503 08:00:14.350754 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet d376ac1ea05c482ca8f03ddf0f5fcd47 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:39891: session id = eae9f53215a34bc88abe7b3c1c223319-d376ac1ea05c482ca8f03ddf0f5fcd47
I20240503 08:00:14.351377 16714 raft_consensus.cc:348] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.351531 13228 tablet_copy_source_session.cc:205] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.351608 16714 raft_consensus.cc:729] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.351784 16714 consensus_queue.cc:260] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.352324 17212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376ac1ea05c482ca8f03ddf0f5fcd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.352705 16714 ts_tablet_manager.cc:1398] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.353160 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-b094d04434d24923806b0e9e209c3fca received from {username='slave'} at 127.0.99.135:44613
I20240503 08:00:14.353235 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-b094d04434d24923806b0e9e209c3fca on tablet b094d04434d24923806b0e9e209c3fca with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.354537 17243 tablet_copy_client.cc:783] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.354701 17243 tablet_copy_client.cc:647] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.356879 17243 tablet_copy_client.cc:523] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.357021 17212 tablet_copy_client.cc:783] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.357141 17212 tablet_copy_client.cc:647] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.358361 17212 tablet_copy_client.cc:523] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.359421 17243 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:14.361884 17271 ts_tablet_manager.cc:902] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:14.362416 17271 tablet_copy_client.cc:308] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:14.362875 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet f1151731adc9407b86d143d6d41f69b4 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:44613)
I20240503 08:00:14.362947 17212 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:14.363001 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet f1151731adc9407b86d143d6d41f69b4 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:44613: session id = b052e89b63c24258872c832a6a0d4d4d-f1151731adc9407b86d143d6d41f69b4
I20240503 08:00:14.363154 16679 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.363481 16679 tablet_bootstrap.cc:492] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:14.363684 16679 ts_tablet_manager.cc:1367] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.170s user 0.026s sys 0.000s
I20240503 08:00:14.363875 13097 tablet_copy_source_session.cc:205] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.364202 16679 raft_consensus.cc:348] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:14.364390 16679 raft_consensus.cc:729] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.364635 17271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1151731adc9407b86d143d6d41f69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.364660 16679 consensus_queue.cc:260] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:14.365433 16679 ts_tablet_manager.cc:1398] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.365842 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-a60b868dcc504bb29d08e2752c8dd44d received from {username='slave'} at 127.0.99.133:36949
I20240503 08:00:14.365912 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-a60b868dcc504bb29d08e2752c8dd44d on tablet a60b868dcc504bb29d08e2752c8dd44d with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:14.369680 17271 tablet_copy_client.cc:783] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.370312 17271 tablet_copy_client.cc:647] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.376235 17271 tablet_copy_client.cc:523] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.377372 12813 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:14.378994 17271 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:14.379048 14658 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.379336 14789 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.379678 12944 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:00:14.379756 12750 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5212dff7ded143a7923bb6176466b0fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:14.380084 15843 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.380208 13206 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.380939 16984 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.381181 16984 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.381404 16984 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.391795 16714 ts_tablet_manager.cc:902] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:14.392256 16714 tablet_copy_client.cc:308] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:14.392648 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 7d61aa445a50417b82916019c2a4298c from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:44613)
I20240503 08:00:14.392740 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 7d61aa445a50417b82916019c2a4298c from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:44613: session id = b052e89b63c24258872c832a6a0d4d4d-7d61aa445a50417b82916019c2a4298c
I20240503 08:00:14.393429 15843 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.393594 13097 tablet_copy_source_session.cc:205] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.394433 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d61aa445a50417b82916019c2a4298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.395351 14658 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.395825 13206 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.399015 12946 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.399094 12667 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.401434 16714 tablet_copy_client.cc:783] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.401609 16714 tablet_copy_client.cc:647] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.404541 16714 tablet_copy_client.cc:523] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.405916 16714 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:14.409302 14789 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:14.420459 16717 ts_tablet_manager.cc:902] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:14.420903 16717 tablet_copy_client.cc:308] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:14.421438 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5480e544dca3428ab4fcf190577d5331 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:14.421545 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5480e544dca3428ab4fcf190577d5331 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-5480e544dca3428ab4fcf190577d5331
I20240503 08:00:14.422449 12835 tablet_copy_source_session.cc:205] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.423286 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5480e544dca3428ab4fcf190577d5331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.423282 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.424697 17212 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.425707 16717 tablet_copy_client.cc:783] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.425853 16717 tablet_copy_client.cc:647] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.430107 14527 raft_consensus.cc:1203] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:14.430948 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.438704 14658 raft_consensus.cc:1203] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.441459 12793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:14.442636 16717 tablet_copy_client.cc:523] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.444144 16717 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:14.447146 17212 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:14.447346 17212 ts_tablet_manager.cc:1367] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.084s user 0.025s sys 0.002s
I20240503 08:00:14.449280 17212 raft_consensus.cc:348] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.449398 17212 raft_consensus.cc:729] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.453065 17212 consensus_queue.cc:260] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.457739 17212 ts_tablet_manager.cc:1398] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:14.458724 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-d376ac1ea05c482ca8f03ddf0f5fcd47 received from {username='slave'} at 127.0.99.136:39891
I20240503 08:00:14.458813 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-d376ac1ea05c482ca8f03ddf0f5fcd47 on tablet d376ac1ea05c482ca8f03ddf0f5fcd47 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:14.462742 17214 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.464537 17243 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.464896 17214 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:14.465150 17243 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:14.465363 17243 ts_tablet_manager.cc:1367] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.106s user 0.027s sys 0.002s
I20240503 08:00:14.465591 17214 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.139s user 0.026s sys 0.000s
I20240503 08:00:14.466132 17243 raft_consensus.cc:348] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.468504 17214 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.468699 17243 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.468794 17214 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.469774 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.470135 17214 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.470333 17243 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:14.471210 14638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.471241 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.471300 17214 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:14.470866 17243 ts_tablet_manager.cc:1398] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:14.473430 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-047aba81fad543ce8b562a184746b163 received from {username='slave'} at 127.0.99.136:39891
I20240503 08:00:14.473506 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-047aba81fad543ce8b562a184746b163 on tablet 047aba81fad543ce8b562a184746b163 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:14.474512 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-72ac9fbe5dfd4a4e89912d7e947eca45 received from {username='slave'} at 127.0.99.135:45337
I20240503 08:00:14.474597 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-72ac9fbe5dfd4a4e89912d7e947eca45 on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.490516 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.511224 16717 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.511547 16717 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:14.511742 16717 ts_tablet_manager.cc:1367] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.068s user 0.025s sys 0.000s
I20240503 08:00:14.512197 16717 raft_consensus.cc:348] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.512331 16717 raft_consensus.cc:729] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.512471 16717 consensus_queue.cc:260] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.512727 16717 ts_tablet_manager.cc:1398] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.513406 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5480e544dca3428ab4fcf190577d5331 received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:14.513486 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5480e544dca3428ab4fcf190577d5331 on tablet 5480e544dca3428ab4fcf190577d5331 with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:14.516516 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.528699 12813 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.530308 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.530700 13206 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.530721 14527 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:00:14.531057 12749 consensus_peers.cc:485] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cee7c94dd0941d99d9dd6ceb39fc190. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:14.531071 14789 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:14.531224 15843 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.531483 15986 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.531728 15986 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.531805 16984 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:14.537127 15843 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.539729 12946 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.541214 13206 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.541777 12667 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.546772 14789 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.550300 14658 raft_consensus.cc:1203] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.555092 14527 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.555495 17271 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.555804 17271 tablet_bootstrap.cc:492] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:14.557307 17271 ts_tablet_manager.cc:1367] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.178s user 0.025s sys 0.000s
I20240503 08:00:14.557755 17271 raft_consensus.cc:348] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.557861 17271 raft_consensus.cc:729] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.557981 17271 consensus_queue.cc:260] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.558290 17271 ts_tablet_manager.cc:1398] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.558748 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-f1151731adc9407b86d143d6d41f69b4 received from {username='slave'} at 127.0.99.135:44613
I20240503 08:00:14.558825 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-f1151731adc9407b86d143d6d41f69b4 on tablet f1151731adc9407b86d143d6d41f69b4 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.563432 16714 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.563679 16714 tablet_bootstrap.cc:492] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:14.563843 16714 ts_tablet_manager.cc:1367] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.158s user 0.022s sys 0.003s
I20240503 08:00:14.564241 16714 raft_consensus.cc:348] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.564347 16714 raft_consensus.cc:729] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.564476 16714 consensus_queue.cc:260] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:14.564754 16714 ts_tablet_manager.cc:1398] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:14.565155 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-7d61aa445a50417b82916019c2a4298c received from {username='slave'} at 127.0.99.135:44613
I20240503 08:00:14.565218 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-7d61aa445a50417b82916019c2a4298c on tablet 7d61aa445a50417b82916019c2a4298c with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.572659 14789 raft_consensus.cc:1203] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.575322 12813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57314
I20240503 08:00:14.575502 12813 raft_consensus.cc:595] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:14.612098 16679 ts_tablet_manager.cc:902] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:14.612569 16679 tablet_copy_client.cc:308] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:14.613086 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5480e544dca3428ab4fcf190577d5331 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:14.613193 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5480e544dca3428ab4fcf190577d5331 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-5480e544dca3428ab4fcf190577d5331
I20240503 08:00:14.614178 12835 tablet_copy_source_session.cc:205] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.616572 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5480e544dca3428ab4fcf190577d5331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.619966 16679 tablet_copy_client.cc:783] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.620124 16679 tablet_copy_client.cc:647] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.621364 16679 tablet_copy_client.cc:523] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.622561 16679 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:14.633121 14920 raft_consensus.cc:1203] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.663901 16679 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:14.664312 16679 tablet_bootstrap.cc:492] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:14.664522 16679 ts_tablet_manager.cc:1367] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.042s user 0.031s sys 0.000s
I20240503 08:00:14.666810 16679 raft_consensus.cc:348] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.667553 16679 raft_consensus.cc:729] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.667728 16679 consensus_queue.cc:260] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.668704 16679 ts_tablet_manager.cc:1398] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:14.669203 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5480e544dca3428ab4fcf190577d5331 received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:14.669283 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5480e544dca3428ab4fcf190577d5331 on tablet 5480e544dca3428ab4fcf190577d5331 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:14.681565 14789 raft_consensus.cc:1203] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.693818 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.696221 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.697016 15843 raft_consensus.cc:979] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: : Instructing follower 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:14.697137 15843 raft_consensus.cc:1067] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Signalling peer 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:14.697822 14658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f"
dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
from {username='slave'} at 127.0.99.129:42235
I20240503 08:00:14.698012 14658 raft_consensus.cc:482] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:14.698124 14658 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:14.699288 14658 raft_consensus.cc:504] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.700675 14658 leader_election.cc:288] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:14.701377 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:14.701503 12813 raft_consensus.cc:3032] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:14.701550 12813 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:14.701705 12813 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.701893 12813 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:14.702909 12813 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:14.704191 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:14.704363 13076 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:14.705417 13076 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:14.705876 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8994c38d0f640e9bd5b5c322c603b9f" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:14.706003 12946 raft_consensus.cc:3037] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:14.706981 12946 raft_consensus.cc:2454] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:14.708596 14594 leader_election.cc:302] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38, 8398e6f89bf5419d81ed5bed991da0cf; no voters:
I20240503 08:00:14.708843 16332 raft_consensus.cc:2781] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:14.709002 16332 raft_consensus.cc:686] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Becoming Leader. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Running, Role: LEADER
I20240503 08:00:14.709333 16332 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:14.710108 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.712409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.712970 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.713593 12664 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: term changed from 1 to 2, leader changed from 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) to 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:14.716336 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.725414 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.729341 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.761477 14920 raft_consensus.cc:1203] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.774765 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.787802 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.799106 14789 raft_consensus.cc:1203] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.801656 14658 raft_consensus.cc:1203] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:14.811950 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.817591 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.825237 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.838791 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.841451 14789 raft_consensus.cc:1203] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.842042 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.845760 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.870152 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.875909 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.876228 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.880260 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.883797 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.887573 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.897693 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.901587 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.905586 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.911592 16717 ts_tablet_manager.cc:902] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:14.912283 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:14.914590 16679 ts_tablet_manager.cc:902] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:14.915058 16679 tablet_copy_client.cc:308] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:14.915474 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5212dff7ded143a7923bb6176466b0fa from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:53111)
I20240503 08:00:14.915568 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5212dff7ded143a7923bb6176466b0fa from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:53111: session id = 82cd33f2bae4419b946095f8eb93e1e4-5212dff7ded143a7923bb6176466b0fa
I20240503 08:00:14.916436 12835 tablet_copy_source_session.cc:205] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.917968 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5212dff7ded143a7923bb6176466b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.918706 14920 raft_consensus.cc:1203] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:14.919219 16717 tablet_copy_client.cc:308] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:14.920215 16679 tablet_copy_client.cc:783] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.920372 16679 tablet_copy_client.cc:647] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.921525 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.922719 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.923166 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:59379)
I20240503 08:00:14.923267 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:59379: session id = 8398e6f89bf5419d81ed5bed991da0cf-5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:14.924150 12835 tablet_copy_source_session.cc:205] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:14.925237 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cee7c94dd0941d99d9dd6ceb39fc190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.927412 16717 tablet_copy_client.cc:783] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:14.927561 16717 tablet_copy_client.cc:647] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:14.930207 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.946117 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:14.953152 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.957530 16679 tablet_copy_client.cc:523] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.959118 16679 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:14.975592 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:14.995167 16717 tablet_copy_client.cc:523] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:14.996629 16717 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:15.010360 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.015084 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.020210 16679 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:15.020502 16679 tablet_bootstrap.cc:492] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:15.020687 16679 ts_tablet_manager.cc:1367] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.062s user 0.023s sys 0.001s
I20240503 08:00:15.021198 16679 raft_consensus.cc:348] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.021333 16679 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.021461 16679 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.022569 16679 ts_tablet_manager.cc:1398] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:15.023399 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5212dff7ded143a7923bb6176466b0fa received from {username='slave'} at 127.0.99.133:53111
I20240503 08:00:15.023475 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-5212dff7ded143a7923bb6176466b0fa on tablet 5212dff7ded143a7923bb6176466b0fa with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:15.027611 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.034605 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.034956 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.041049 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.044214 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.044705 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.069519 14527 raft_consensus.cc:1203] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:15.101619 12813 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:15.102226 17566 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.118511 16717 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:15.119038 16717 tablet_bootstrap.cc:492] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:15.119392 16717 ts_tablet_manager.cc:1367] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.123s user 0.027s sys 0.000s
I20240503 08:00:15.119957 16717 raft_consensus.cc:348] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:15.120162 16717 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.120365 16717 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:15.120752 16717 ts_tablet_manager.cc:1398] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.121258 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5cee7c94dd0941d99d9dd6ceb39fc190 received from {username='slave'} at 127.0.99.134:59379
I20240503 08:00:15.121335 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-5cee7c94dd0941d99d9dd6ceb39fc190 on tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:15.124773 13208 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.126170 13076 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.128512 15689 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.128841 14527 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.130568 16772 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.136950 17666 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.138461 14527 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.140920 12666 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.142127 13076 consensus_queue.cc:237] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:15.143462 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c5de226c7a4b349eef86a79d172851 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:15.143800 14658 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:15.144165 14920 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:15.144296 16848 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:15.144399 13208 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:15.144990 16848 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:15.145253 16848 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:15.146864 17020 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:15.147243 14920 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:15.149034 14658 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:15.150249 12666 catalog_manager.cc:5484] T 672fc9981d074f7e8f2324ba0b06be57 P eae9f53215a34bc88abe7b3c1c223319 reported cstate change: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:15.153558 13208 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:15.154767 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 672fc9981d074f7e8f2324ba0b06be57 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:15.155510 17710 tablet_replica.cc:330] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:15.155956 17710 raft_consensus.cc:2229] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:15.156234 17710 raft_consensus.cc:2258] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:15.157330 17710 ts_tablet_manager.cc:1875] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:15.159894 17710 ts_tablet_manager.cc:1888] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:15.160156 17710 log.cc:1196] T 672fc9981d074f7e8f2324ba0b06be57 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/672fc9981d074f7e8f2324ba0b06be57
I20240503 08:00:15.161072 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 672fc9981d074f7e8f2324ba0b06be57 (table rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]) successfully deleted
I20240503 08:00:15.161568 13076 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.163268 17701 tablet_replica.cc:330] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:15.163532 17701 raft_consensus.cc:2229] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:15.163699 17701 raft_consensus.cc:2258] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:15.164656 17701 ts_tablet_manager.cc:1875] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:15.167295 17701 ts_tablet_manager.cc:1888] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:15.167440 17701 log.cc:1196] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/70c5de226c7a4b349eef86a79d172851
I20240503 08:00:15.168047 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 70c5de226c7a4b349eef86a79d172851 (table rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]) successfully deleted
I20240503 08:00:15.238529 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.238958 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.242117 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.250056 13076 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:15.250726 17566 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.253698 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.260143 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.262290 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.288780 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.295758 14658 raft_consensus.cc:1203] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:15.302549 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.321892 12944 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:15.322527 17566 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.357755 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.360775 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.367622 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.377131 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.398424 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.400127 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33786
I20240503 08:00:15.400295 13208 raft_consensus.cc:595] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:15.401399 14527 raft_consensus.cc:1203] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:15.406998 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.426731 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.444515 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.457525 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.457988 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.465922 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.482587 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.486737 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.507285 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.509199 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.511690 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.512300 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.513587 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.525014 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.529103 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.533183 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.535177 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.551443 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.558123 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.561992 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.566965 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.585155 13188 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:15.592909 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.593910 14507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.607270 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.612860 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.620810 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.655637 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33844
I20240503 08:00:15.655807 13208 raft_consensus.cc:595] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:15.668406 17666 raft_consensus.cc:979] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: : Instructing follower 82cd33f2bae4419b946095f8eb93e1e4 to start an election
I20240503 08:00:15.668534 17666 raft_consensus.cc:1067] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer 82cd33f2bae4419b946095f8eb93e1e4 to start an election
I20240503 08:00:15.669677 14527 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851"
dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
from {username='slave'} at 127.0.99.132:48927
I20240503 08:00:15.669883 14527 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:15.669977 14527 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.671020 14527 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.672048 14527 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [CANDIDATE]: Term 2 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:15.673120 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "82cd33f2bae4419b946095f8eb93e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:15.673262 13076 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.674082 13076 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd33f2bae4419b946095f8eb93e1e4 in term 2.
I20240503 08:00:15.674842 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "82cd33f2bae4419b946095f8eb93e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:15.674998 13208 raft_consensus.cc:3032] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:15.675048 13208 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:15.675031 14460 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 82cd33f2bae4419b946095f8eb93e1e4; no voters:
I20240503 08:00:15.675191 13208 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.675355 16331 raft_consensus.cc:2781] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:15.675371 13208 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.675515 16331 raft_consensus.cc:686] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Running, Role: LEADER
I20240503 08:00:15.675791 16331 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.676352 13208 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd33f2bae4419b946095f8eb93e1e4 in term 2.
I20240503 08:00:15.678853 12668 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.725047 12944 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:15.726886 13076 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.727033 14658 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.727339 16784 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.728109 16784 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.731459 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.736373 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.745244 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.749135 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.751497 13208 consensus_queue.cc:237] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:15.752838 12944 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.753975 17666 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.755824 17666 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.758226 12944 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.760320 12672 catalog_manager.cc:5484] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.760818 14527 raft_consensus.cc:1261] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:15.761371 17666 consensus_queue.cc:1035] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:15.762545 14527 raft_consensus.cc:2932] T d44b2c67014742af9a65a3576f509702 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:15.765789 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.766268 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet d44b2c67014742af9a65a3576f509702 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:15.769814 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.775480 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.786082 16784 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:15.786594 18077 tablet_replica.cc:330] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:15.786911 18077 raft_consensus.cc:2229] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:15.787456 18077 raft_consensus.cc:2258] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:15.788372 13076 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:15.788857 18077 ts_tablet_manager.cc:1875] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:15.791828 12672 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:15.792059 14658 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:15.792292 18077 ts_tablet_manager.cc:1888] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:15.792362 18077 log.cc:1196] T d44b2c67014742af9a65a3576f509702 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d44b2c67014742af9a65a3576f509702
I20240503 08:00:15.793959 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet d44b2c67014742af9a65a3576f509702 (table rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]) successfully deleted
I20240503 08:00:15.801618 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7f229898c24572815177628720521f with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:15.813134 13188 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:15.835311 15689 raft_consensus.cc:979] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:15.835424 15689 raft_consensus.cc:1067] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:15.835855 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.132:52351
I20240503 08:00:15.836057 14789 raft_consensus.cc:482] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:15.836158 14789 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.837198 14789 raft_consensus.cc:504] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:15.838408 14789 leader_election.cc:288] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:15.840150 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:15.840260 13076 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.840608 18091 tablet_replica.cc:330] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:15.840806 18091 raft_consensus.cc:2229] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:15.840955 18091 raft_consensus.cc:2258] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:15.841246 13076 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:15.841637 18091 ts_tablet_manager.cc:1875] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:15.843756 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:15.843866 12813 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.844671 12813 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:15.846259 14724 leader_election.cc:302] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38, b052e89b63c24258872c832a6a0d4d4d; no voters:
I20240503 08:00:15.848737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.849081 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c630e5cf7c0419cbf5d13413d22d317" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:15.849195 13206 raft_consensus.cc:3032] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:15.849256 13206 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:15.849386 13206 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:15.849574 13206 raft_consensus.cc:3037] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:15.850467 13206 raft_consensus.cc:2454] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:15.858891 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.860318 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.863023 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:15.880156 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:15.887758 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.892593 14769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.894074 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:15.917193 18120 raft_consensus.cc:2781] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:15.917377 18120 raft_consensus.cc:686] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:15.917728 18120 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:15.843793 18091 ts_tablet_manager.cc:1888] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:15.919087 18091 log.cc:1196] T 9e7f229898c24572815177628720521f P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e7f229898c24572815177628720521f
I20240503 08:00:15.920087 12669 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:15.920374 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 9e7f229898c24572815177628720521f (table rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]) successfully deleted
I20240503 08:00:15.987730 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:15.995579 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.000591 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.009150 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.015247 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.027513 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.028004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.083755 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.084888 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.095666 13076 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.096277 17997 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.099704 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.101363 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.104249 13208 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.105129 16331 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.110631 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.122751 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.138336 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.152261 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.186121 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.190336 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.228376 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.231971 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.232204 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.239436 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.239663 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.243522 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.248608 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.261524 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.263654 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.267066 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.271909 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.292168 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.306437 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.309278 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.310503 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.346417 14638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.346702 12793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.349037 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.354736 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.357348 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.381587 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.383015 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.383271 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.420598 12813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57502
I20240503 08:00:16.420737 12813 raft_consensus.cc:595] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:16.423895 13208 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:16.424491 18139 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.434070 12813 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:16.434568 18139 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.438021 13076 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:16.439091 18139 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.460803 13076 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:16.462256 14920 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.462406 13208 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.462754 17020 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.463922 17020 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.465282 13076 consensus_queue.cc:237] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:16.465327 17020 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:16.465682 14920 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:16.467022 14788 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.469573 17020 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.469934 13208 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:16.471390 12668 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:16.472851 12813 raft_consensus.cc:1261] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.473235 18413 consensus_queue.cc:1035] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.474071 16848 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.474457 14788 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.475313 12813 raft_consensus.cc:2932] T f1151731adc9407b86d143d6d41f69b4 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.479106 12672 catalog_manager.cc:5484] T f1151731adc9407b86d143d6d41f69b4 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.481086 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 076781af2ac14ddc87d31e3ec77db08f with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:16.483019 18433 tablet_replica.cc:330] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:16.483590 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet f1151731adc9407b86d143d6d41f69b4 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:16.485329 18433 raft_consensus.cc:2229] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:16.485749 18433 raft_consensus.cc:2258] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:16.486325 15843 raft_consensus.cc:979] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: : Instructing follower 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:16.486342 18433 ts_tablet_manager.cc:1875] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:16.486413 15843 raft_consensus.cc:1067] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Signalling peer 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:16.486840 14658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3"
dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
from {username='slave'} at 127.0.99.129:42235
I20240503 08:00:16.487005 14658 raft_consensus.cc:482] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:16.487063 14658 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:16.488090 14658 raft_consensus.cc:504] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:16.488737 14658 leader_election.cc:288] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:16.489444 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:16.489475 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:16.489569 13208 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:16.489609 13076 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:16.490417 13076 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:16.490608 13208 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:16.491180 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:16.491322 12813 raft_consensus.cc:3032] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:16.491364 12813 raft_consensus.cc:729] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:16.491506 12813 consensus_queue.cc:260] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:16.491727 12813 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:16.492753 12813 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:16.493525 14593 leader_election.cc:302] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 6c358da9faef48c4adb12cce628aeb38, 8398e6f89bf5419d81ed5bed991da0cf; no voters:
I20240503 08:00:16.493703 17842 raft_consensus.cc:2781] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:16.493854 17842 raft_consensus.cc:686] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Becoming Leader. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Running, Role: LEADER
I20240503 08:00:16.494010 17842 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:16.495905 12673 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: term changed from 1 to 2, leader changed from 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) to 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:16.498723 18433 ts_tablet_manager.cc:1888] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:16.498808 18433 log.cc:1196] T 076781af2ac14ddc87d31e3ec77db08f P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/076781af2ac14ddc87d31e3ec77db08f
I20240503 08:00:16.499379 18433 tablet_replica.cc:330] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:16.499560 18433 raft_consensus.cc:2229] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:16.499733 18433 raft_consensus.cc:2258] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:16.500319 18433 ts_tablet_manager.cc:1875] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:16.500546 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 076781af2ac14ddc87d31e3ec77db08f (table rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]) successfully deleted
I20240503 08:00:16.505986 18433 ts_tablet_manager.cc:1888] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:16.506091 18433 log.cc:1196] T f1151731adc9407b86d143d6d41f69b4 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1151731adc9407b86d143d6d41f69b4
I20240503 08:00:16.506661 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet f1151731adc9407b86d143d6d41f69b4 (table rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]) successfully deleted
I20240503 08:00:16.507782 14527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a2990625964c0e8b79780890b0e6c3" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
I20240503 08:00:16.508000 14527 raft_consensus.cc:3037] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:16.508998 14527 raft_consensus.cc:2454] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:16.521345 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.526130 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.526676 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.541831 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.545893 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.546113 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:16.554239 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.596118 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:16.618273 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:16.619921 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.619935 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.623512 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.627880 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.642351 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.642834 12813 consensus_queue.cc:237] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:16.644230 14527 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.644229 13208 raft_consensus.cc:1261] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:16.644623 15843 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.644920 15843 consensus_queue.cc:1035] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:16.649374 15986 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:16.651955 14527 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:16.652233 12672 catalog_manager.cc:5484] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:16.654546 13208 raft_consensus.cc:2932] T a924487225dc4246aeb695060084096b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:16.655647 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet a924487225dc4246aeb695060084096b with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:16.656003 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.658582 18527 tablet_replica.cc:330] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:16.658779 18527 raft_consensus.cc:2229] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:16.659173 18527 raft_consensus.cc:2258] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:16.660336 18527 ts_tablet_manager.cc:1875] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:16.663780 18527 ts_tablet_manager.cc:1888] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:16.663910 18527 log.cc:1196] T a924487225dc4246aeb695060084096b P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a924487225dc4246aeb695060084096b
I20240503 08:00:16.664430 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet a924487225dc4246aeb695060084096b (table rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]) successfully deleted
I20240503 08:00:16.677650 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:16.739663 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.754309 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.758626 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:16.774679 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.782519 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:16.785571 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:16.789520 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.793423 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.796449 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.815112 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.817741 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:16.824532 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.828792 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.830184 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.830829 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.849171 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.870189 14789 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:16.871754 12813 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.872224 13076 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.872938 18388 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.873199 18388 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.875196 18388 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.880092 12813 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.882434 13076 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.883894 12671 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:16.886606 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:16.887101 18646 tablet_replica.cc:330] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:16.887450 18646 raft_consensus.cc:2229] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:16.887694 18646 raft_consensus.cc:2258] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:16.888837 18646 ts_tablet_manager.cc:1875] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:16.894515 18646 ts_tablet_manager.cc:1888] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:00:16.894641 18646 log.cc:1196] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:16.895247 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 7c630e5cf7c0419cbf5d13413d22d317 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:00:16.915097 12944 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e7f229898c24572815177628720521f"
dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54436
I20240503 08:00:16.915268 12944 raft_consensus.cc:595] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:16.918201 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:16.926173 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:16.937839 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:16.941817 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.946853 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:16.948216 14527 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.948599 18441 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.957055 12813 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.957461 18441 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.965119 13076 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.965626 13208 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:16.966857 17842 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.967191 17842 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:16.972440 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:16.972491 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:16.983577 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.025003 12944 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:17.026470 14527 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:17.027233 18743 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:17.028380 12813 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:17.029428 18745 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:17.033205 18743 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.033712 14527 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.035503 12813 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.038370 12670 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.039453 12813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57576
I20240503 08:00:17.039584 12813 raft_consensus.cc:595] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:17.041774 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:17.041992 18527 tablet_replica.cc:330] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:17.042124 18527 raft_consensus.cc:2229] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:17.042260 18527 raft_consensus.cc:2258] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:17.042857 18527 ts_tablet_manager.cc:1875] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:17.045197 18527 ts_tablet_manager.cc:1888] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:17.045261 18527 log.cc:1196] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dcdde0a31f7d4fd0bf20c894f215e3b6
I20240503 08:00:17.045691 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet dcdde0a31f7d4fd0bf20c894f215e3b6 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:00:17.059324 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.060566 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.066052 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.085484 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.086010 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:17.113155 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.118147 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.140655 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:17.171692 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:17.178624 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.179102 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:17.191885 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.193483 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.202284 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.204229 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.204241 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.207976 12924 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:17.213770 18745 raft_consensus.cc:979] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: : Instructing follower 39827620daed48cb8f137442e1f1da76 to start an election
I20240503 08:00:17.214820 18745 raft_consensus.cc:1067] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Signalling peer 39827620daed48cb8f137442e1f1da76 to start an election
I20240503 08:00:17.215649 13076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e7f229898c24572815177628720521f"
dest_uuid: "39827620daed48cb8f137442e1f1da76"
from {username='slave'} at 127.0.99.130:48673
I20240503 08:00:17.215803 13076 raft_consensus.cc:482] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:17.215854 13076 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.216931 13076 raft_consensus.cc:504] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:17.217692 13076 leader_election.cc:288] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 2 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:17.218448 14658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
I20240503 08:00:17.218608 14658 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.219123 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7f229898c24572815177628720521f" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:17.219226 12944 raft_consensus.cc:3032] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:17.219259 12944 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:17.219390 12944 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:17.219534 12944 raft_consensus.cc:3037] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.220417 12944 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 2.
I20240503 08:00:17.220882 13013 leader_election.cc:302] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:00:17.221155 18414 raft_consensus.cc:2781] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:17.221246 18414 raft_consensus.cc:686] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:17.221375 18414 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:17.221765 14658 raft_consensus.cc:2454] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 2.
I20240503 08:00:17.224318 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.228235 12668 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 1 to 2, leader changed from 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.229820 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.230120 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:17.237772 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.244221 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.264400 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.299525 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.359759 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.365715 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.367244 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.367470 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:17.387763 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.403689 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.410028 14769 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.415191 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.415329 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.418983 15986 raft_consensus.cc:979] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:17.419061 15986 raft_consensus.cc:1067] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:17.419412 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.129:57591
I20240503 08:00:17.419528 14789 raft_consensus.cc:482] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:17.419575 14789 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.420534 14789 raft_consensus.cc:504] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:17.420943 14789 leader_election.cc:288] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:17.421355 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:17.421456 12813 raft_consensus.cc:3032] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:17.421489 12813 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:17.421602 12813 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:17.421759 12813 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.422616 12813 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:17.423606 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:17.423740 13206 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.424605 13206 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:17.424993 14722 leader_election.cc:302] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84, b052e89b63c24258872c832a6a0d4d4d; no voters:
I20240503 08:00:17.425159 18392 raft_consensus.cc:2781] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:17.425285 18392 raft_consensus.cc:686] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:17.425412 18392 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:17.426702 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:17.426829 13076 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:17.427127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:17.427778 13076 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:17.428507 12668 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:17.429415 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.453866 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.454850 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.472208 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.485328 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.494959 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.525638 14657 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:17.527180 12813 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:17.527464 12944 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:17.527601 18703 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.528389 18983 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.530364 18703 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.532521 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.533263 12668 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 4 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.534039 12813 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.534353 12944 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.537225 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet d8994c38d0f640e9bd5b5c322c603b9f with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:17.537428 18527 tablet_replica.cc:330] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:17.537554 18527 raft_consensus.cc:2229] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:17.537688 18527 raft_consensus.cc:2258] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:17.538137 18527 ts_tablet_manager.cc:1875] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:17.540129 18527 ts_tablet_manager.cc:1888] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:00:17.540201 18527 log.cc:1196] T d8994c38d0f640e9bd5b5c322c603b9f P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d8994c38d0f640e9bd5b5c322c603b9f
I20240503 08:00:17.540691 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet d8994c38d0f640e9bd5b5c322c603b9f (table rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]) successfully deleted
I20240503 08:00:17.561012 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.566659 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.568970 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.569772 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.575084 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.583884 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.584451 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:17.620379 13076 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:17.621526 14919 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.621893 18413 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.623777 18414 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:17.626109 12672 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:17.629415 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 076781af2ac14ddc87d31e3ec77db08f with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:17.629614 13076 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:17.629832 19044 tablet_replica.cc:330] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:17.630007 19044 raft_consensus.cc:2229] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:17.630478 19044 raft_consensus.cc:2258] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:17.630719 14920 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:17.630844 13011 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 076781af2ac14ddc87d31e3ec77db08f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:17.631083 19044 ts_tablet_manager.cc:1875] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:17.631587 14919 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:17.633141 18413 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.633775 19044 ts_tablet_manager.cc:1888] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:17.633884 19044 log.cc:1196] T 076781af2ac14ddc87d31e3ec77db08f P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/076781af2ac14ddc87d31e3ec77db08f
I20240503 08:00:17.634661 13206 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:17.634850 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 076781af2ac14ddc87d31e3ec77db08f (table rebalance_test_table_5 [id=3ede3e1269b3418f8ad902769287e993]) successfully deleted
I20240503 08:00:17.636159 14527 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.636296 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.636610 15689 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.636875 14919 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.637238 18414 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.637893 15689 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.638180 15689 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.638585 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 076781af2ac14ddc87d31e3ec77db08f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:17.639837 14920 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.639863 12672 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 6 to 7, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:17.644433 15689 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.644737 13077 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.645736 14919 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.647404 12672 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.647388 14527 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.649988 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cc823a5ce34916894cfde0a6614bd1 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:17.651314 19063 tablet_replica.cc:330] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:17.651515 19063 raft_consensus.cc:2229] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:17.651733 19063 raft_consensus.cc:2258] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:17.652253 19063 ts_tablet_manager.cc:1875] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:17.654635 19063 ts_tablet_manager.cc:1888] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:17.654754 19063 log.cc:1196] T 20cc823a5ce34916894cfde0a6614bd1 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/20cc823a5ce34916894cfde0a6614bd1
I20240503 08:00:17.655858 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 20cc823a5ce34916894cfde0a6614bd1 (table rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]) successfully deleted
I20240503 08:00:17.732453 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.734265 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.744514 14657 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:17.744786 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.745071 19045 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.765995 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.767553 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.770324 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.770608 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.809535 12813 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:17.810711 12944 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:17.810844 14789 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.810879 14527 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.811395 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:17.811359 15986 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.811619 18414 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.811687 15986 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.811914 13077 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:17.815135 15843 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:17.827572 15986 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.827980 13077 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.830415 14789 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.830404 14527 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.830543 12665 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:17.837905 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:17.838145 19063 tablet_replica.cc:330] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:17.838287 19063 raft_consensus.cc:2229] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:17.838439 19063 raft_consensus.cc:2258] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:17.839047 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:17.839097 19063 ts_tablet_manager.cc:1875] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:17.841183 19063 ts_tablet_manager.cc:1888] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:17.841241 19063 log.cc:1196] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:17.841706 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 389b40c2136d45ab86ccf49df8d1b71e (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:17.848007 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.848246 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.855199 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.857919 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.862828 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.863943 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:17.865831 13077 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:17.866326 18963 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:17.866792 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.873549 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:17.883241 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.883692 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.894220 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.899415 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:17.927968 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:17.960769 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:17.968359 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:17.988236 12815 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:17.988740 18963 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:17.999029 13206 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:18.001422 18963 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.021867 13206 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:18.023180 12815 consensus_queue.cc:237] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:18.023521 14657 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.023926 19061 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.024649 12813 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.025307 14789 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.025465 12944 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.025836 15843 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.026535 19061 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.027062 19214 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.027348 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.027490 14657 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.029350 19061 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.029456 19214 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.031917 14657 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.033388 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.033421 12813 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.034348 12670 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.038069 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3e4066cd14aa18619310a4490246b with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:18.038250 18527 tablet_replica.cc:330] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:18.038364 18527 raft_consensus.cc:2229] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.038475 18527 raft_consensus.cc:2258] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.038964 18527 ts_tablet_manager.cc:1875] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.041079 18527 ts_tablet_manager.cc:1888] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:18.041141 18527 log.cc:1196] T 02d3e4066cd14aa18619310a4490246b P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/02d3e4066cd14aa18619310a4490246b
I20240503 08:00:18.044091 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 02d3e4066cd14aa18619310a4490246b (table rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]) successfully deleted
I20240503 08:00:18.047930 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.052076 15843 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.054327 14789 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.055223 14658 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.058010 12672 catalog_manager.cc:5484] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.061287 12944 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.064271 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:18.064473 19044 tablet_replica.cc:330] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:18.064620 19044 raft_consensus.cc:2229] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.064759 19044 raft_consensus.cc:2258] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.065367 19044 ts_tablet_manager.cc:1875] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.070986 19044 ts_tablet_manager.cc:1888] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:18.071064 19044 log.cc:1196] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6ab3fc1d0e6a42f38406bb75f69b8cbc
I20240503 08:00:18.071535 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc (table rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]) successfully deleted
I20240503 08:00:18.074568 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.074656 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.074831 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.079077 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.086037 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.151247 14658 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:18.152757 14527 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:18.153613 12813 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:18.153890 18986 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.154204 18986 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.155297 13077 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:18.157979 19385 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.164054 13077 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.163753 19385 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.167439 12667 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.167690 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.170102 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a2990625964c0e8b79780890b0e6c3 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:18.173713 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.177482 19044 tablet_replica.cc:330] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:18.177635 19044 raft_consensus.cc:2229] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.177807 19044 raft_consensus.cc:2258] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.178074 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.178335 19044 ts_tablet_manager.cc:1875] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.181994 19044 ts_tablet_manager.cc:1888] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:18.182070 19044 log.cc:1196] T d9a2990625964c0e8b79780890b0e6c3 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9a2990625964c0e8b79780890b0e6c3
I20240503 08:00:18.182785 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet d9a2990625964c0e8b79780890b0e6c3 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
I20240503 08:00:18.185891 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.192510 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.193143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.202205 19409 ts_tablet_manager.cc:902] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:18.202780 19409 tablet_copy_client.cc:308] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:18.203333 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 076781af2ac14ddc87d31e3ec77db08f from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:55423)
I20240503 08:00:18.203433 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 076781af2ac14ddc87d31e3ec77db08f from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:55423: session id = 8398e6f89bf5419d81ed5bed991da0cf-076781af2ac14ddc87d31e3ec77db08f
I20240503 08:00:18.203855 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.204308 13097 tablet_copy_source_session.cc:205] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:18.205320 19409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:18.208855 19409 tablet_copy_client.cc:783] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:18.209115 19409 tablet_copy_client.cc:647] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:18.210311 12813 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.210983 19409 tablet_copy_client.cc:523] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:18.212404 19409 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:18.214085 14527 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.214468 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.242900 19409 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:18.243450 19409 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:18.243644 19409 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.002s
I20240503 08:00:18.244131 19409 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:18.244254 19409 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:18.244433 19409 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:18.244724 19409 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:18.245160 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-076781af2ac14ddc87d31e3ec77db08f received from {username='slave'} at 127.0.99.134:55423
I20240503 08:00:18.245230 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-076781af2ac14ddc87d31e3ec77db08f on tablet 076781af2ac14ddc87d31e3ec77db08f with peer 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:18.263239 13077 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:18.265110 14919 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:18.265573 14658 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:18.265794 14527 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:18.265792 18413 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.266433 12944 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:18.266644 18413 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.266942 19045 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.267477 19045 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:18.272212 19045 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.272529 14919 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.274297 14658 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.275813 12667 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.280813 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet a60b868dcc504bb29d08e2752c8dd44d with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:18.280879 12946 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.288458 19451 tablet_replica.cc:330] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:18.289903 19451 raft_consensus.cc:2229] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.292870 14527 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.297066 19451 raft_consensus.cc:2258] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.304682 19451 ts_tablet_manager.cc:1875] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.316020 19451 ts_tablet_manager.cc:1888] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:18.316185 19451 log.cc:1196] T a60b868dcc504bb29d08e2752c8dd44d P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a60b868dcc504bb29d08e2752c8dd44d
I20240503 08:00:18.316555 12946 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:18.318029 13206 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.318029 14919 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.318414 18743 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.318754 18743 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.319424 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet a60b868dcc504bb29d08e2752c8dd44d (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:18.329928 18743 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:18.332129 14919 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:18.334415 13208 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:18.336896 12665 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 reported cstate change: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:18.341063 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4f2a993164fb6b1df674311698006 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:18.341249 19451 tablet_replica.cc:330] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:18.341367 19451 raft_consensus.cc:2229] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.341488 19451 raft_consensus.cc:2258] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.342048 19451 ts_tablet_manager.cc:1875] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.345296 19451 ts_tablet_manager.cc:1888] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:18.345386 19451 log.cc:1196] T 48e4f2a993164fb6b1df674311698006 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/48e4f2a993164fb6b1df674311698006
I20240503 08:00:18.345882 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 48e4f2a993164fb6b1df674311698006 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:18.348939 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.363016 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.371071 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.371948 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.374620 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.377764 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.379169 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.393605 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.400604 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.400609 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.407506 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.434378 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.438349 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.444090 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.446557 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.474457 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.515847 12813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57820
I20240503 08:00:18.516000 12813 raft_consensus.cc:595] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:18.549432 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.549547 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.561118 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.561374 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.564049 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.575922 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.581950 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.584277 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.590750 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.601591 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.604460 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.617357 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.621769 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.638382 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.664012 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.668673 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.698069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:18.698128 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.699257 14658 raft_consensus.cc:1203] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:18.700220 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.700354 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.728163 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.732724 19204 raft_consensus.cc:979] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:18.732837 19204 raft_consensus.cc:1067] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:18.733335 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.129:57591
I20240503 08:00:18.733466 14789 raft_consensus.cc:482] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:18.733518 14789 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.734594 14789 raft_consensus.cc:504] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:18.735309 14789 leader_election.cc:288] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:18.737363 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:18.737483 12813 raft_consensus.cc:3032] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:18.737527 12813 raft_consensus.cc:729] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:18.737658 12813 consensus_queue.cc:260] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:18.737891 12813 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.738859 12813 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:18.740258 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:18.740375 13208 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.740505 12813 consensus_queue.cc:237] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:18.740566 14769 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.741369 13208 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:18.741732 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.742295 13208 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.742630 15986 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.743839 14658 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.744242 19204 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.745114 14526 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.745630 19204 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.753849 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.755460 14658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
I20240503 08:00:18.755571 14658 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.756597 14658 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:18.757158 14723 leader_election.cc:302] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, 8398e6f89bf5419d81ed5bed991da0cf, af712db2f85249adbe718e8ba0549f84, b052e89b63c24258872c832a6a0d4d4d; no voters:
I20240503 08:00:18.757371 19271 raft_consensus.cc:2781] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:18.757527 19271 raft_consensus.cc:686] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:18.757683 19271 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:18.760088 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:18.760216 12944 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.760331 12673 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:18.761169 12944 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:18.762768 14527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5212dff7ded143a7923bb6176466b0fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
I20240503 08:00:18.762882 14527 raft_consensus.cc:3037] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:18.763455 19702 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.763886 14527 raft_consensus.cc:2454] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:18.765519 13208 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.766729 14658 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.768654 12672 catalog_manager.cc:5484] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.771905 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 5480e544dca3428ab4fcf190577d5331 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:18.772225 19717 tablet_replica.cc:330] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:18.772411 19717 raft_consensus.cc:2229] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.772943 19717 raft_consensus.cc:2258] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.773506 19717 ts_tablet_manager.cc:1875] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.775624 19717 ts_tablet_manager.cc:1888] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:18.775739 19717 log.cc:1196] T 5480e544dca3428ab4fcf190577d5331 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5480e544dca3428ab4fcf190577d5331
I20240503 08:00:18.776237 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 5480e544dca3428ab4fcf190577d5331 (table rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]) successfully deleted
I20240503 08:00:18.792714 14526 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:18.805056 13077 consensus_queue.cc:237] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:18.806872 14919 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.807251 12944 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.807809 14789 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:18.808251 19440 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.810348 18414 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.810643 18414 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:18.822682 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.826236 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:18.828428 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:18.836323 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:18.838414 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:18.838682 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:18.855672 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:18.858588 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:18.872721 19440 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.875551 12667 catalog_manager.cc:5484] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.876569 12944 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.880939 14919 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.883615 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d61aa445a50417b82916019c2a4298c with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:18.886889 14789 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.898766 13208 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:18.900364 12813 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.900810 19049 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.902009 14789 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:18.902369 19785 tablet_replica.cc:330] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:18.902499 19785 raft_consensus.cc:2229] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.902581 19049 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:18.902658 19785 raft_consensus.cc:2258] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.903421 19785 ts_tablet_manager.cc:1875] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.906075 19785 ts_tablet_manager.cc:1888] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:18.906196 19785 log.cc:1196] T 7d61aa445a50417b82916019c2a4298c P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d61aa445a50417b82916019c2a4298c
I20240503 08:00:18.907128 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 7d61aa445a50417b82916019c2a4298c (table rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]) successfully deleted
I20240503 08:00:18.907997 19763 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.909983 14789 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.915566 12813 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.915863 12672 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:18.919330 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7bf40aa0614923a19feed8bab430fa with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:18.944096 19801 tablet_replica.cc:330] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:18.944339 19801 raft_consensus.cc:2229] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:18.944561 19801 raft_consensus.cc:2258] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:18.947537 19801 ts_tablet_manager.cc:1875] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:18.950021 19801 ts_tablet_manager.cc:1888] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:18.953965 19801 log.cc:1196] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c7bf40aa0614923a19feed8bab430fa
I20240503 08:00:18.954640 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 9c7bf40aa0614923a19feed8bab430fa (table rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]) successfully deleted
I20240503 08:00:18.998526 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.009056 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.012410 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.013810 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.016492 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.019098 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.026069 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.056967 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.063503 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.065476 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.070402 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.072115 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.074635 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.083907 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.105520 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.115608 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.197659 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.204959 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.207266 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.218272 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.228993 14526 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:19.229480 19711 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.237929 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.241462 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.269280 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.275641 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.278402 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.299633 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.304281 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.309769 12946 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:19.310295 19711 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.314329 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.315786 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.316049 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.320876 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.323546 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.337664 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.341614 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.350975 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.351720 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.352263 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.353062 12815 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:19.354513 19711 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.355993 13208 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:19.356555 19711 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.369041 14658 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:19.369832 19711 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.373508 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.395350 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.415927 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.429275 13077 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:19.430938 14658 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:19.431710 14526 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:19.431695 19777 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.432114 14919 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:19.432261 19777 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.432492 19735 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.437180 19778 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:19.439630 14658 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:19.441892 14526 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:19.443994 12671 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:19.450785 14919 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:19.461840 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet a60b868dcc504bb29d08e2752c8dd44d with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:19.465351 20114 tablet_replica.cc:330] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:19.465550 20114 raft_consensus.cc:2229] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:19.465785 20114 raft_consensus.cc:2258] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.466385 20114 ts_tablet_manager.cc:1875] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:19.468839 20114 ts_tablet_manager.cc:1888] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:19.468951 20114 log.cc:1196] T a60b868dcc504bb29d08e2752c8dd44d P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a60b868dcc504bb29d08e2752c8dd44d
I20240503 08:00:19.469772 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet a60b868dcc504bb29d08e2752c8dd44d (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:19.483475 13077 consensus_queue.cc:237] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:19.485841 12946 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:19.486799 14789 raft_consensus.cc:1261] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:19.487706 19735 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:19.487928 19778 consensus_queue.cc:1035] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:19.493008 19778 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.493760 12946 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.495221 14789 raft_consensus.cc:2932] T b094d04434d24923806b0e9e209c3fca P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.497400 12671 catalog_manager.cc:5484] T b094d04434d24923806b0e9e209c3fca P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.503050 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet b094d04434d24923806b0e9e209c3fca with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:19.509305 12815 consensus_queue.cc:237] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:19.510762 14658 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:19.511170 15986 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.511210 14526 raft_consensus.cc:1261] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:19.511595 15986 consensus_queue.cc:1035] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:19.513298 19204 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.514189 14658 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.514755 14526 raft_consensus.cc:2932] T 5480e544dca3428ab4fcf190577d5331 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.517647 12671 catalog_manager.cc:5484] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.518118 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.520452 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 5480e544dca3428ab4fcf190577d5331 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:19.522156 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.525655 20162 tablet_replica.cc:330] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:19.525828 20162 raft_consensus.cc:2229] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:19.525983 20162 raft_consensus.cc:2258] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.526521 20162 ts_tablet_manager.cc:1875] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:19.530467 20162 ts_tablet_manager.cc:1888] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:19.530606 20162 log.cc:1196] T b094d04434d24923806b0e9e209c3fca P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b094d04434d24923806b0e9e209c3fca
I20240503 08:00:19.535305 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.538574 13208 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:19.541261 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.541733 14658 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.542117 19049 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.542738 13077 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.542748 14789 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.543279 19061 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.543720 19061 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.543998 20174 tablet_replica.cc:330] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:19.544129 20174 raft_consensus.cc:2229] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:19.544294 20174 raft_consensus.cc:2258] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.545125 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet b094d04434d24923806b0e9e209c3fca (table rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]) successfully deleted
I20240503 08:00:19.546299 20174 ts_tablet_manager.cc:1875] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:19.548594 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.549185 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.552956 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.555471 20174 ts_tablet_manager.cc:1888] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:19.555548 20174 log.cc:1196] T 5480e544dca3428ab4fcf190577d5331 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5480e544dca3428ab4fcf190577d5331
I20240503 08:00:19.563129 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 5480e544dca3428ab4fcf190577d5331 (table rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]) successfully deleted
I20240503 08:00:19.584996 19061 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.585532 14658 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.586333 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.587306 14789 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.588430 12665 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.591040 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c84f7a57e0945d9a25a70798f054bd1 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:19.591274 20114 tablet_replica.cc:330] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:19.591432 20114 raft_consensus.cc:2229] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:19.591583 20114 raft_consensus.cc:2258] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.592191 20114 ts_tablet_manager.cc:1875] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:19.594372 20114 ts_tablet_manager.cc:1888] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:19.594445 20114 log.cc:1196] T 1c84f7a57e0945d9a25a70798f054bd1 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c84f7a57e0945d9a25a70798f054bd1
I20240503 08:00:19.594442 13077 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:19.595425 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 1c84f7a57e0945d9a25a70798f054bd1 (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:19.661203 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.664142 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.672322 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.675076 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.675395 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.676503 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.677938 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.707378 13077 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:19.709275 14919 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.709720 19778 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.710367 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.711619 19778 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.712538 14789 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:19.713773 19778 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:19.731125 19735 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.731770 14919 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.736150 12671 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P eae9f53215a34bc88abe7b3c1c223319 reported cstate change: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.736680 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.740770 14789 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:19.743008 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet a45748a98c3447d3aa2e8de94981a745 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:19.743328 20174 tablet_replica.cc:330] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:19.743458 20174 raft_consensus.cc:2229] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:19.743592 20174 raft_consensus.cc:2258] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.744187 20174 ts_tablet_manager.cc:1875] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:19.746440 20174 ts_tablet_manager.cc:1888] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:19.746515 20174 log.cc:1196] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a45748a98c3447d3aa2e8de94981a745
I20240503 08:00:19.747344 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet a45748a98c3447d3aa2e8de94981a745 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:00:19.749318 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.781231 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.798519 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.798519 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.799317 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.799568 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.800616 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.801055 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.817095 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.832083 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.839864 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.845990 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.849646 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.853572 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.875459 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:19.882601 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.884212 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.885928 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:19.887472 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:19.891055 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.896859 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.903725 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.905700 13188 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.913611 12793 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.924477 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:19.994102 14789 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:19.995707 13208 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:19.996305 14526 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:19.996510 12815 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:19.996558 20068 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.996672 14658 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:19.996881 20068 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.997524 20068 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.997790 20068 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:19.999521 20068 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.001127 12815 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.001960 13208 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.002602 14658 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.003301 14526 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.004537 12672 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 6 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.008677 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 5212dff7ded143a7923bb6176466b0fa with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:20.008893 20114 tablet_replica.cc:330] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:20.009008 20114 raft_consensus.cc:2229] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.009173 20114 raft_consensus.cc:2258] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.010183 20114 ts_tablet_manager.cc:1875] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.012601 20114 ts_tablet_manager.cc:1888] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:00:20.012727 20114 log.cc:1196] T 5212dff7ded143a7923bb6176466b0fa P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5212dff7ded143a7923bb6176466b0fa
I20240503 08:00:20.013394 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 5212dff7ded143a7923bb6176466b0fa (table rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]) successfully deleted
I20240503 08:00:20.028573 13208 consensus_queue.cc:237] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:20.030025 13077 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:20.031965 12815 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.033150 14919 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.033315 14658 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.034071 19049 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.034601 13206 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.034960 19777 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.035804 19049 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.036849 19778 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.038676 19777 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.041864 14658 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.043828 12667 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.047537 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:20.047799 20114 tablet_replica.cc:330] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:20.047936 20114 raft_consensus.cc:2229] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.048096 20114 raft_consensus.cc:2258] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.048635 20114 ts_tablet_manager.cc:1875] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.048764 13206 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.050807 20114 ts_tablet_manager.cc:1888] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:20.051218 20114 log.cc:1196] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e0d84a47cc34af9bdd6b955feb2c3ba
I20240503 08:00:20.051801 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 0e0d84a47cc34af9bdd6b955feb2c3ba (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:20.054873 13206 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "047aba81fad543ce8b562a184746b163"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34390
I20240503 08:00:20.055038 13206 raft_consensus.cc:595] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:20.059403 19049 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:20.061815 12813 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:20.064289 12671 catalog_manager.cc:5484] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:20.069197 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:20.069449 20114 tablet_replica.cc:330] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:20.069608 20114 raft_consensus.cc:2229] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.069800 20114 raft_consensus.cc:2258] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.070434 20114 ts_tablet_manager.cc:1875] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.071867 14919 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:20.072940 20114 ts_tablet_manager.cc:1888] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:00:20.073024 20114 log.cc:1196] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d376ac1ea05c482ca8f03ddf0f5fcd47
I20240503 08:00:20.074278 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet d376ac1ea05c482ca8f03ddf0f5fcd47 (table rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]) successfully deleted
I20240503 08:00:20.117271 19049 raft_consensus.cc:979] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: : Instructing follower 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:00:20.117372 19049 raft_consensus.cc:1067] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:00:20.117830 12813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "047aba81fad543ce8b562a184746b163"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
from {username='slave'} at 127.0.99.132:60853
I20240503 08:00:20.117983 12813 raft_consensus.cc:482] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:20.118057 12813 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:20.119179 12813 raft_consensus.cc:504] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:20.120298 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:20.120414 13206 raft_consensus.cc:3032] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:20.120456 13206 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:20.120579 13206 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:20.120735 13206 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:20.121712 13206 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 2.
I20240503 08:00:20.123309 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:20.123425 12944 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:20.124197 12944 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 2.
I20240503 08:00:20.124526 12751 leader_election.cc:302] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:20.124711 19204 raft_consensus.cc:2781] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:20.124919 19204 raft_consensus.cc:686] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:20.125129 19204 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:20.126933 12813 leader_election.cc:288] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 2 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:20.127923 12665 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:20.132050 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047aba81fad543ce8b562a184746b163" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eae9f53215a34bc88abe7b3c1c223319"
I20240503 08:00:20.132241 14919 raft_consensus.cc:3037] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:20.133389 14919 raft_consensus.cc:2454] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 2.
I20240503 08:00:20.151307 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.152217 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.173300 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.178251 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.204847 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.205930 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.206806 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.241050 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.246562 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.251709 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.256763 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.277490 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.286301 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.292315 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.295344 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.301766 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.345554 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.345554 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.376884 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.385577 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.392262 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.392760 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.404917 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.432431 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.483844 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.492878 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.499118 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.504101 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.512784 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.517163 12944 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:20.518189 20528 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.524868 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.530651 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.541909 12813 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:20.544700 14527 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.545617 14789 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:20.545917 19204 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.546301 14788 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.546797 19204 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.547319 14658 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.551820 19204 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.553623 12813 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:20.554070 20068 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.557719 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.558347 20528 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.564663 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.569020 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.572582 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.579690 20528 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.582767 12671 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.584965 12946 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.587945 14788 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.588409 14527 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.589135 20068 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:20.589440 14658 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.592561 12667 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:20.593180 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:20.593689 12813 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:20.596439 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:20.603678 20729 tablet_replica.cc:330] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:20.603880 20729 raft_consensus.cc:2229] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.604498 20729 raft_consensus.cc:2258] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.605072 20729 ts_tablet_manager.cc:1875] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.606354 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.606834 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.608836 20729 ts_tablet_manager.cc:1888] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:20.608940 20729 log.cc:1196] T 5cee7c94dd0941d99d9dd6ceb39fc190 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:20.609891 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
I20240503 08:00:20.609674 14789 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:20.612725 12813 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:20.613128 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.613400 20728 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:20.616052 12813 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.617861 20068 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.618425 20731 tablet_replica.cc:330] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:20.618582 20731 raft_consensus.cc:2229] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.618754 20731 raft_consensus.cc:2258] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.619203 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.619810 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7c630e5cf7c0419cbf5d13413d22d317 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:20.619952 20731 ts_tablet_manager.cc:1875] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.620697 12667 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 7 to 8, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.624437 20731 ts_tablet_manager.cc:1888] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:20.625633 20731 log.cc:1196] T 7c630e5cf7c0419cbf5d13413d22d317 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:20.626260 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 7c630e5cf7c0419cbf5d13413d22d317 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
W20240503 08:00:20.629261 14724 consensus_peers.cc:485] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c630e5cf7c0419cbf5d13413d22d317. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.642324 14919 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:20.643146 20711 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.647686 13206 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:20.648967 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.654328 20528 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.661479 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.685226 13206 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:20.687253 14789 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.687280 14658 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.687700 20189 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.687994 19049 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.690547 12813 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:20.691529 19049 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.692204 14526 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.692226 14788 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:20.692862 19204 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.693182 19204 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:20.693851 14789 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.695019 12667 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.696236 20528 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.697969 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c84f7a57e0945d9a25a70798f054bd1 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:20.698201 20731 tablet_replica.cc:330] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:20.698082 14658 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:20.698330 20731 raft_consensus.cc:2229] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.698505 20731 raft_consensus.cc:2258] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.699105 20731 ts_tablet_manager.cc:1875] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.698982 14526 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.701198 20731 ts_tablet_manager.cc:1888] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:20.701261 20731 log.cc:1196] T 1c84f7a57e0945d9a25a70798f054bd1 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c84f7a57e0945d9a25a70798f054bd1
I20240503 08:00:20.701932 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 1c84f7a57e0945d9a25a70798f054bd1 (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:20.702955 12665 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.706651 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:20.709416 20731 tablet_replica.cc:330] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:20.709794 20731 raft_consensus.cc:2229] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:20.709998 20731 raft_consensus.cc:2258] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:20.710700 20731 ts_tablet_manager.cc:1875] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:20.713040 20731 ts_tablet_manager.cc:1888] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:20.713158 20731 log.cc:1196] T 389b40c2136d45ab86ccf49df8d1b71e P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:20.713654 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 389b40c2136d45ab86ccf49df8d1b71e (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:20.722127 14788 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:20.769886 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.772325 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.772333 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.776726 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.800567 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.813263 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.815312 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.823258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.911550 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.911837 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.917182 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:20.918334 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.937415 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.938903 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:20.951768 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.952416 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:20.955083 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.961269 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.965373 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.970685 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:20.977205 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:20.994951 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.007731 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.021392 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.027138 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.032298 13208 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:21.034332 14919 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.034816 19049 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.035444 14526 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.036628 12946 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.037060 20786 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.037405 20786 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.038513 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:21.047340 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.061942 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.072994 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.082511 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.089881 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:21.091226 19049 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:21.091632 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.094568 14526 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:21.096872 12672 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:21.102677 14919 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:21.114495 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet c917c5849f54411eb55d69ab33456bf4 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:21.114717 20731 tablet_replica.cc:330] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:21.114831 20731 raft_consensus.cc:2229] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.115358 20731 raft_consensus.cc:2258] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.115942 20731 ts_tablet_manager.cc:1875] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.118357 20731 ts_tablet_manager.cc:1888] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:21.118415 20731 log.cc:1196] T c917c5849f54411eb55d69ab33456bf4 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c917c5849f54411eb55d69ab33456bf4
I20240503 08:00:21.118899 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet c917c5849f54411eb55d69ab33456bf4 (table rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]) successfully deleted
I20240503 08:00:21.125401 12946 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:21.134616 12946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6"
dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54884
I20240503 08:00:21.134765 12946 raft_consensus.cc:595] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:21.183162 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.192893 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.200318 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.204722 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.223307 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.223740 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.250876 21112 ts_tablet_manager.cc:902] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:21.250906 19505 raft_consensus.cc:979] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: : Instructing follower 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:00:21.251072 19505 raft_consensus.cc:1067] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Signalling peer 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:00:21.251421 21112 tablet_copy_client.cc:308] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:21.251668 12813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
from {username='slave'} at 127.0.99.130:56231
I20240503 08:00:21.251792 12813 raft_consensus.cc:482] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:21.251837 12813 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:21.252817 12813 raft_consensus.cc:504] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.253168 12813 leader_election.cc:288] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 2 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:21.253984 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:21.254093 12946 raft_consensus.cc:3032] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:21.254127 12946 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:21.254259 12946 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.254405 12946 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:21.255478 12946 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 2.
I20240503 08:00:21.255739 14526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdde0a31f7d4fd0bf20c894f215e3b6" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
I20240503 08:00:21.255863 14526 raft_consensus.cc:3037] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:21.256850 14526 raft_consensus.cc:2454] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 2.
I20240503 08:00:21.257402 12750 leader_election.cc:302] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, 82cd33f2bae4419b946095f8eb93e1e4; no voters:
I20240503 08:00:21.257575 20528 raft_consensus.cc:2781] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:21.257719 20528 raft_consensus.cc:686] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:21.257862 20528 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.257882 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:58731)
I20240503 08:00:21.258009 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:58731: session id = 82cd33f2bae4419b946095f8eb93e1e4-7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:21.259836 14808 tablet_copy_source_session.cc:205] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.259860 12670 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 1 to 2, leader changed from 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.261317 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.264078 21112 tablet_copy_client.cc:783] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.264235 21112 tablet_copy_client.cc:647] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.265728 21112 tablet_copy_client.cc:523] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.265913 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.267038 21112 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:00:21.272897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.297262 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.298058 21112 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.298380 21112 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:00:21.298569 21112 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.000s
I20240503 08:00:21.299053 21112 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.299171 21112 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:00:21.299480 21112 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.299775 21112 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:21.300451 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-7c630e5cf7c0419cbf5d13413d22d317 received from {username='slave'} at 127.0.99.133:58731
I20240503 08:00:21.300546 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-7c630e5cf7c0419cbf5d13413d22d317 on tablet 7c630e5cf7c0419cbf5d13413d22d317 with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:21.301684 14788 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:21.303519 13077 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:21.303699 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:21.304136 20728 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.305277 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.306046 20728 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.309475 20728 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:21.312767 12670 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 4 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:21.314184 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.317219 13208 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:21.317414 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:21.324138 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.327621 13077 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:21.331764 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.333245 21164 tablet_replica.cc:330] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:21.333454 21164 raft_consensus.cc:2229] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.333632 21164 raft_consensus.cc:2258] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.334447 21164 ts_tablet_manager.cc:1875] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.338728 21164 ts_tablet_manager.cc:1888] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:00:21.338876 21164 log.cc:1196] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/72ac9fbe5dfd4a4e89912d7e947eca45
I20240503 08:00:21.339553 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:21.344425 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.362249 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.369254 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.430295 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.433424 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.440524 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.449023 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.452625 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.463733 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.464542 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.466905 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.468235 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.468847 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.469920 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.491379 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.492374 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.495602 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.512106 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.554805 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.588780 14526 raft_consensus.cc:1203] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:21.592767 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.606434 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.607353 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.610277 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.624155 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.639686 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.641299 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.643240 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.678776 14658 consensus_queue.cc:237] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.680352 13077 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:21.681180 14526 raft_consensus.cc:1261] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:21.681195 19382 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:21.682307 19382 consensus_queue.cc:1035] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:21.685335 19382 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.687948 13077 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.691699 12664 catalog_manager.cc:5484] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 7 to 8, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.692140 14526 raft_consensus.cc:2932] T d9a2990625964c0e8b79780890b0e6c3 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.696952 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a2990625964c0e8b79780890b0e6c3 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:21.697345 21164 tablet_replica.cc:330] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:21.697513 21164 raft_consensus.cc:2229] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.697657 21164 raft_consensus.cc:2258] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.698292 21164 ts_tablet_manager.cc:1875] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.699527 12813 consensus_queue.cc:237] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:21.700714 21164 ts_tablet_manager.cc:1888] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:00:21.700779 21164 log.cc:1196] T d9a2990625964c0e8b79780890b0e6c3 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9a2990625964c0e8b79780890b0e6c3
I20240503 08:00:21.700986 14658 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:21.701043 14789 raft_consensus.cc:1261] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:21.701488 20528 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:21.702231 20528 consensus_queue.cc:1035] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:21.703734 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet d9a2990625964c0e8b79780890b0e6c3 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
I20240503 08:00:21.713059 21148 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:21.714227 14789 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:21.716892 12671 catalog_manager.cc:5484] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 6 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:21.726519 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:21.727491 14658 raft_consensus.cc:2932] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:21.729257 12946 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:21.729995 20528 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.732751 14526 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:21.733222 21148 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.737401 21378 tablet_replica.cc:330] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:21.737604 21378 raft_consensus.cc:2229] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.737844 21378 raft_consensus.cc:2258] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.738453 21378 ts_tablet_manager.cc:1875] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.742347 21378 ts_tablet_manager.cc:1888] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:21.744655 14789 consensus_queue.cc:237] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:21.746100 14526 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:21.746131 14658 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:21.746248 13208 raft_consensus.cc:1261] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:21.746507 20728 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:21.746842 20728 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:21.747150 20728 consensus_queue.cc:1035] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:21.751032 21378 log.cc:1196] T 6ab3fc1d0e6a42f38406bb75f69b8cbc P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ab3fc1d0e6a42f38406bb75f69b8cbc
I20240503 08:00:21.750995 20728 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.751770 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc (table rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]) successfully deleted
I20240503 08:00:21.753187 14526 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.755394 14658 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.755419 13208 raft_consensus.cc:2932] T 5212dff7ded143a7923bb6176466b0fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.755816 12670 catalog_manager.cc:5484] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:21.761183 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5212dff7ded143a7923bb6176466b0fa with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:21.761411 21164 tablet_replica.cc:330] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:21.761572 21164 raft_consensus.cc:2229] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.761715 21164 raft_consensus.cc:2258] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.762171 21164 ts_tablet_manager.cc:1875] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.764757 21164 ts_tablet_manager.cc:1888] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 08:00:21.765074 21164 log.cc:1196] T 5212dff7ded143a7923bb6176466b0fa P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5212dff7ded143a7923bb6176466b0fa
I20240503 08:00:21.765719 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 5212dff7ded143a7923bb6176466b0fa (table rebalance_test_table_8 [id=e36f346a18884285a63bee81e275ca47]) successfully deleted
I20240503 08:00:21.800820 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.804569 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.821877 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.824978 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.825405 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.827800 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.839798 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.840754 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:21.842245 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.853013 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.863662 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.890993 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.892712 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:21.900115 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:21.912470 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:21.926841 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.009899 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.013796 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.018741 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:22.025702 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.028112 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.039126 14658 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:22.040441 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.040829 12946 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:22.041302 21332 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.048074 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.057341 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.065840 21332 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:22.068619 12946 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:22.072661 12665 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 6 to 7, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:22.079051 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet d8994c38d0f640e9bd5b5c322c603b9f with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:22.079255 21164 tablet_replica.cc:330] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:22.079380 21164 raft_consensus.cc:2229] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.079485 21164 raft_consensus.cc:2258] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.079985 21164 ts_tablet_manager.cc:1875] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.082298 21164 ts_tablet_manager.cc:1888] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:22.083390 21164 log.cc:1196] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d8994c38d0f640e9bd5b5c322c603b9f
I20240503 08:00:22.084193 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet d8994c38d0f640e9bd5b5c322c603b9f (table rebalance_test_table_1 [id=cc180bf9c7ae466f9b724fe299385926]) successfully deleted
I20240503 08:00:22.094594 14658 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.097849 12946 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:22.098434 21332 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.100662 21332 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.102396 12658 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d8994c38d0f640e9bd5b5c322c603b9f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:22.103107 12946 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.106225 12665 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
W20240503 08:00:22.119127 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8994c38d0f640e9bd5b5c322c603b9f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.154448 12813 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.156244 12946 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:22.156971 20528 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.157133 14919 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:22.157572 20528 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.162285 20528 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.164728 14919 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.164986 12946 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.169255 12665 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 4 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.172291 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 047aba81fad543ce8b562a184746b163 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:22.174398 21744 tablet_replica.cc:330] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:22.174726 21744 raft_consensus.cc:2229] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.174861 21744 raft_consensus.cc:2258] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.179623 21744 ts_tablet_manager.cc:1875] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.182993 21744 ts_tablet_manager.cc:1888] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 08:00:22.183221 21744 log.cc:1196] T 047aba81fad543ce8b562a184746b163 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/047aba81fad543ce8b562a184746b163
I20240503 08:00:22.183795 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 047aba81fad543ce8b562a184746b163 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
I20240503 08:00:22.204658 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.212026 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.215001 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.219028 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.223490 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.224337 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.239472 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:22.284358 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.308727 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.319960 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.322110 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.325206 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.327217 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.336949 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:22.339943 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51642
I20240503 08:00:22.340106 13208 raft_consensus.cc:595] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:22.346328 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.389854 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.401301 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.422849 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.436228 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.439072 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.452072 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.453260 13188 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.465210 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.478020 14769 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:22.524003 21026 raft_consensus.cc:979] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:22.524114 21026 raft_consensus.cc:1067] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:22.524684 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.132:52351
I20240503 08:00:22.524829 14789 raft_consensus.cc:482] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:22.524896 14789 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.525947 14789 raft_consensus.cc:504] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:22.526381 14789 leader_election.cc:288] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:22.527318 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:22.527451 12813 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.528529 12813 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:22.528972 14724 leader_election.cc:302] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, b052e89b63c24258872c832a6a0d4d4d; no voters:
I20240503 08:00:22.529189 21419 raft_consensus.cc:2781] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:22.529369 21419 raft_consensus.cc:686] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:22.529531 21419 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:22.530956 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:22.531095 13208 raft_consensus.cc:3032] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:22.531136 13208 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:22.531260 13208 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:22.531404 13208 raft_consensus.cc:3037] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.531795 12670 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:22.532276 13208 raft_consensus.cc:2454] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:22.533479 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.543220 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.544062 13188 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:22.574280 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:22.588749 14526 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:22.590111 13077 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:22.590641 18526 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.595691 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.613209 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.615422 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.624650 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.632207 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.632102 18526 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:22.632493 13077 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:22.634035 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.635404 12672 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:22.640404 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.640870 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c5de226c7a4b349eef86a79d172851 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:22.641115 21744 tablet_replica.cc:330] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:22.641245 21744 raft_consensus.cc:2229] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.641346 21744 raft_consensus.cc:2258] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.641971 21744 ts_tablet_manager.cc:1875] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.644294 21744 ts_tablet_manager.cc:1888] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:22.644367 21744 log.cc:1196] T 70c5de226c7a4b349eef86a79d172851 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/70c5de226c7a4b349eef86a79d172851
I20240503 08:00:22.647127 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 70c5de226c7a4b349eef86a79d172851 (table rebalance_test_table_2 [id=ecd7ad5de8d5489baabee552c3a8e2ae]) successfully deleted
I20240503 08:00:22.656811 13188 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:22.661370 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.670200 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.676142 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.679224 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.685176 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:22.686059 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:22.689525 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.690901 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.704756 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.719944 14526 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.722587 13077 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:22.723294 22016 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.725397 22016 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.727705 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 70c5de226c7a4b349eef86a79d172851 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:22.728185 12666 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.728324 13077 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.731412 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.736383 22025 ts_tablet_manager.cc:902] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
W20240503 08:00:22.737192 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70c5de226c7a4b349eef86a79d172851. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.737690 22025 tablet_copy_client.cc:308] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39483
I20240503 08:00:22.755116 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c7bf40aa0614923a19feed8bab430fa"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51674
I20240503 08:00:22.755288 13208 raft_consensus.cc:595] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:00:22.755348 13208 raft_consensus.cc:603] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:00:22.757169 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.758610 14678 tablet_copy_service.cc:140] P 8398e6f89bf5419d81ed5bed991da0cf: Received BeginTabletCopySession request for tablet d8994c38d0f640e9bd5b5c322c603b9f from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:36667)
I20240503 08:00:22.758750 14678 tablet_copy_service.cc:161] P 8398e6f89bf5419d81ed5bed991da0cf: Beginning new tablet copy session on tablet d8994c38d0f640e9bd5b5c322c603b9f from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:36667: session id = eae9f53215a34bc88abe7b3c1c223319-d8994c38d0f640e9bd5b5c322c603b9f
I20240503 08:00:22.760268 14678 tablet_copy_source_session.cc:205] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.761898 22025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.764127 22025 tablet_copy_client.cc:783] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.764276 22025 tablet_copy_client.cc:647] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.768764 22025 tablet_copy_client.cc:523] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.770313 22025 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:22.778762 13188 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:22.805135 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51680
I20240503 08:00:22.805284 13208 raft_consensus.cc:595] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:22.813967 22025 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.814469 22025 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:22.814783 22025 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.004s
I20240503 08:00:22.815451 22025 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.815809 22025 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:22.816016 22025 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.818048 22025 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:22.819303 14678 tablet_copy_service.cc:342] P 8398e6f89bf5419d81ed5bed991da0cf: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-d8994c38d0f640e9bd5b5c322c603b9f received from {username='slave'} at 127.0.99.136:36667
I20240503 08:00:22.819414 14678 tablet_copy_service.cc:433] P 8398e6f89bf5419d81ed5bed991da0cf: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-d8994c38d0f640e9bd5b5c322c603b9f on tablet d8994c38d0f640e9bd5b5c322c603b9f with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:22.840979 12946 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.842352 14919 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.842835 21133 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.847097 21133 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.847555 14919 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.850217 12666 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 reported cstate change: config changed from index 5 to 6, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.852269 21327 raft_consensus.cc:979] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: : Instructing follower 0cf63591cfc244028e1fc368c2bca079 to start an election
I20240503 08:00:22.852361 21327 raft_consensus.cc:1067] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer 0cf63591cfc244028e1fc368c2bca079 to start an election
I20240503 08:00:22.852911 12946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4"
dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
from {username='slave'} at 127.0.99.132:40315
I20240503 08:00:22.853056 12946 raft_consensus.cc:482] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:22.853112 12946 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.854184 12946 raft_consensus.cc:504] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.854702 12946 leader_election.cc:288] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 2 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311), eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:22.854981 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4f2a993164fb6b1df674311698006 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:22.855021 14526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
I20240503 08:00:22.855157 14526 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.855185 21744 tablet_replica.cc:330] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:22.855299 21744 raft_consensus.cc:2229] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.855445 21744 raft_consensus.cc:2258] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.855985 21744 ts_tablet_manager.cc:1875] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.856223 14526 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 2.
I20240503 08:00:22.858193 21744 ts_tablet_manager.cc:1888] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:22.858259 21744 log.cc:1196] T 48e4f2a993164fb6b1df674311698006 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/48e4f2a993164fb6b1df674311698006
I20240503 08:00:22.859166 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 48e4f2a993164fb6b1df674311698006 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:22.859766 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:22.859885 13208 raft_consensus.cc:3032] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:22.859921 13208 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:22.860070 13208 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.860091 12946 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:22.860252 13208 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.861244 13208 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 2.
I20240503 08:00:22.861582 12879 leader_election.cc:302] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 82cd33f2bae4419b946095f8eb93e1e4, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:22.862021 21133 raft_consensus.cc:2781] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:22.862107 21133 raft_consensus.cc:686] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 LEADER]: Becoming Leader. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:22.862159 14920 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.862234 21133 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:22.863106 21133 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.863444 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c917c5849f54411eb55d69ab33456bf4" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "eae9f53215a34bc88abe7b3c1c223319"
I20240503 08:00:22.863559 14919 raft_consensus.cc:3037] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:22.864351 12666 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:22.865655 14919 raft_consensus.cc:2454] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf63591cfc244028e1fc368c2bca079 in term 2.
I20240503 08:00:22.869010 21133 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:22.871767 12660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 48e4f2a993164fb6b1df674311698006 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:22.871762 14919 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:22.874522 12672 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 reported cstate change: config changed from index 6 to 7, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
W20240503 08:00:22.907032 12881 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e4f2a993164fb6b1df674311698006. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.923028 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.931902 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:22.936239 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.944535 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.955169 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:22.967128 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.969868 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:22.975658 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:22.982827 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.996747 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.017529 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.025079 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.027856 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.028173 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.041771 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.048449 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.051537 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.065570 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.069041 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.083088 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.084506 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.091342 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.091430 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.110713 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.116000 12815 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:23.116922 21959 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.125072 13208 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:23.127100 21959 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.147259 22025 ts_tablet_manager.cc:902] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:23.147758 22025 tablet_copy_client.cc:308] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38311
I20240503 08:00:23.152357 14547 tablet_copy_service.cc:140] P 82cd33f2bae4419b946095f8eb93e1e4: Received BeginTabletCopySession request for tablet 70c5de226c7a4b349eef86a79d172851 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:47589)
I20240503 08:00:23.152491 14547 tablet_copy_service.cc:161] P 82cd33f2bae4419b946095f8eb93e1e4: Beginning new tablet copy session on tablet 70c5de226c7a4b349eef86a79d172851 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:47589: session id = eae9f53215a34bc88abe7b3c1c223319-70c5de226c7a4b349eef86a79d172851
I20240503 08:00:23.153892 14547 tablet_copy_source_session.cc:205] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.155249 22025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.157490 22025 tablet_copy_client.cc:783] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.157646 22025 tablet_copy_client.cc:647] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.161936 22025 tablet_copy_client.cc:523] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.163225 22025 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:23.198792 22025 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.199090 22025 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:23.199317 22025 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.002s
I20240503 08:00:23.199806 22025 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:23.199932 22025 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:23.200093 22025 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:23.200533 22025 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:23.201243 14547 tablet_copy_service.cc:342] P 82cd33f2bae4419b946095f8eb93e1e4: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-70c5de226c7a4b349eef86a79d172851 received from {username='slave'} at 127.0.99.136:47589
I20240503 08:00:23.201346 14547 tablet_copy_service.cc:433] P 82cd33f2bae4419b946095f8eb93e1e4: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-70c5de226c7a4b349eef86a79d172851 on tablet 70c5de226c7a4b349eef86a79d172851 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:23.232247 13077 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.233530 14657 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.234133 22024 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.241027 22024 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:23.243933 12672 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:23.246995 14657 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:23.249722 13077 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
W20240503 08:00:23.251299 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e7f229898c24572815177628720521f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.251710 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7f229898c24572815177628720521f with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:23.252846 14657 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:23.255301 22024 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.264072 19735 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.267181 12672 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.267381 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9e7f229898c24572815177628720521f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:23.267829 14657 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.276885 22354 tablet_replica.cc:330] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:23.277107 22354 raft_consensus.cc:2229] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.277241 22354 raft_consensus.cc:2258] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.277776 22354 ts_tablet_manager.cc:1875] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.284680 22354 ts_tablet_manager.cc:1888] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:23.284775 22354 log.cc:1196] T 9e7f229898c24572815177628720521f P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e7f229898c24572815177628720521f
I20240503 08:00:23.285260 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 9e7f229898c24572815177628720521f (table rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]) successfully deleted
I20240503 08:00:23.311584 14919 raft_consensus.cc:1203] T d8994c38d0f640e9bd5b5c322c603b9f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:23.365015 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.365931 13208 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.366706 22130 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.371744 14526 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.373754 22130 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.376392 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.381129 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.388278 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.388711 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.389999 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.395190 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.398747 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.398787 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.399111 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.408283 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.408566 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.430070 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.434994 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.443101 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.443243 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.444428 14919 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.445899 22130 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.449537 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.453838 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.457114 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.461246 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.468958 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.497289 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.508672 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.513747 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.521437 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.526425 22509 ts_tablet_manager.cc:902] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:00:23.530505 22509 tablet_copy_client.cc:308] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33937
I20240503 08:00:23.530941 12966 tablet_copy_service.cc:140] P 0cf63591cfc244028e1fc368c2bca079: Received BeginTabletCopySession request for tablet 48e4f2a993164fb6b1df674311698006 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:48127)
I20240503 08:00:23.531044 12966 tablet_copy_service.cc:161] P 0cf63591cfc244028e1fc368c2bca079: Beginning new tablet copy session on tablet 48e4f2a993164fb6b1df674311698006 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:48127: session id = b052e89b63c24258872c832a6a0d4d4d-48e4f2a993164fb6b1df674311698006
I20240503 08:00:23.531963 12966 tablet_copy_source_session.cc:205] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.532183 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.535126 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.535903 22509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.537324 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.546286 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.546510 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.548280 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.561901 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.591536 22509 tablet_copy_client.cc:783] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.591722 22509 tablet_copy_client.cc:647] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.593490 22509 tablet_copy_client.cc:523] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.600147 22509 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:00:23.620734 14919 raft_consensus.cc:1203] T 70c5de226c7a4b349eef86a79d172851 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:23.644671 22509 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.644847 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51792
I20240503 08:00:23.644990 13208 raft_consensus.cc:595] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:23.645190 22509 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:00:23.645440 22509 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.004s
I20240503 08:00:23.645998 22509 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:23.646184 22509 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:00:23.646387 22509 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:23.648479 22509 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:23.650471 12966 tablet_copy_service.cc:342] P 0cf63591cfc244028e1fc368c2bca079: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-48e4f2a993164fb6b1df674311698006 received from {username='slave'} at 127.0.99.135:48127
I20240503 08:00:23.650553 12966 tablet_copy_service.cc:433] P 0cf63591cfc244028e1fc368c2bca079: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-48e4f2a993164fb6b1df674311698006 on tablet 48e4f2a993164fb6b1df674311698006 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:23.656942 12815 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53604
I20240503 08:00:23.657089 12815 raft_consensus.cc:595] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:23.679737 14789 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:23.681901 13208 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 08:00:23.682269 14725 consensus_peers.cc:485] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.682292 22225 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.683089 12815 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.683193 14789 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:23.683642 22225 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.684376 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:23.685092 22225 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.685570 22225 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:23.685904 12815 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:23.688213 12672 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:23.693560 13208 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:23.695693 22610 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:23.697276 13208 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:23.704659 12672 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 6 to 7, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:23.711042 14789 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:23.712682 13057 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36122
I20240503 08:00:23.713105 22671 tablet_replica.cc:330] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:00:23.713289 22671 raft_consensus.cc:2229] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.713657 22671 raft_consensus.cc:2258] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.714296 22671 ts_tablet_manager.cc:1875] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.715020 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:23.715375 22610 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.718262 22671 ts_tablet_manager.cc:1888] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:23.718377 22671 log.cc:1196] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/72ac9fbe5dfd4a4e89912d7e947eca45
W20240503 08:00:23.718953 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ac9fbe5dfd4a4e89912d7e947eca45. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.719537 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:00:23.723096 22617 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.724617 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:23.725808 12666 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.726801 13206 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:23.764668 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.769685 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.772532 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.781167 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.788120 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.799671 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.802338 22717 ts_tablet_manager.cc:902] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:00:23.810732 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.812541 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.812942 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.816378 22717 tablet_copy_client.cc:308] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:00:23.820302 14638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.820600 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 9e7f229898c24572815177628720521f from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:50817)
I20240503 08:00:23.820694 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 9e7f229898c24572815177628720521f from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:50817: session id = eae9f53215a34bc88abe7b3c1c223319-9e7f229898c24572815177628720521f
I20240503 08:00:23.821583 13097 tablet_copy_source_session.cc:205] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.827323 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.828491 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.829386 22717 tablet_copy_client.cc:783] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.829524 22717 tablet_copy_client.cc:647] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.831323 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.841311 22717 tablet_copy_client.cc:523] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.842657 22717 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:23.844552 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.871980 22717 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.872285 22717 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:23.872469 22717 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.000s
I20240503 08:00:23.872944 22717 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:23.873078 22717 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:23.873240 22717 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:23.874220 22717 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:23.875042 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-9e7f229898c24572815177628720521f received from {username='slave'} at 127.0.99.136:50817
I20240503 08:00:23.875140 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-9e7f229898c24572815177628720521f on tablet 9e7f229898c24572815177628720521f with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:23.879597 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.880247 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.887897 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.893314 21327 raft_consensus.cc:979] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: : Instructing follower 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:23.893402 21327 raft_consensus.cc:1067] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer 8398e6f89bf5419d81ed5bed991da0cf to start an election
I20240503 08:00:23.899204 14657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b"
dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
from {username='slave'} at 127.0.99.132:51043
I20240503 08:00:23.899343 14657 raft_consensus.cc:482] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:23.899398 14657 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.900399 14657 raft_consensus.cc:504] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.900774 14657 leader_election.cc:288] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:23.901319 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:23.901427 12815 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.902307 12815 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:23.902474 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:23.902529 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.903746 14593 leader_election.cc:302] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, 8398e6f89bf5419d81ed5bed991da0cf; no voters:
I20240503 08:00:23.904083 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3e4066cd14aa18619310a4490246b" candidate_uuid: "8398e6f89bf5419d81ed5bed991da0cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:23.904197 13206 raft_consensus.cc:3032] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:23.904248 13206 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:23.904379 19382 raft_consensus.cc:2781] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:23.904369 13206 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.904500 13206 raft_consensus.cc:3037] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.904508 19382 raft_consensus.cc:686] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Becoming Leader. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Running, Role: LEADER
I20240503 08:00:23.904642 19382 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.905362 13206 raft_consensus.cc:2454] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8398e6f89bf5419d81ed5bed991da0cf in term 2.
I20240503 08:00:23.907447 12671 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134). New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:23.910643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.920938 14769 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:23.922111 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:23.926349 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:23.931332 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.932169 22535 raft_consensus.cc:979] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:23.932242 22535 raft_consensus.cc:1067] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:23.932627 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.129:57591
I20240503 08:00:23.932744 14789 raft_consensus.cc:482] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:23.932798 14789 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.933789 14789 raft_consensus.cc:504] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.934230 14789 leader_election.cc:288] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311), 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:23.934973 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
I20240503 08:00:23.935109 14657 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.936054 14657 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:23.937003 12815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
I20240503 08:00:23.937111 12815 raft_consensus.cc:3032] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:23.937144 12815 raft_consensus.cc:729] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:00:23.937266 12815 consensus_queue.cc:260] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.937448 12815 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.938351 12815 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:23.938709 14526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
I20240503 08:00:23.938830 14526 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.939819 14526 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:23.940287 14724 leader_election.cc:302] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, 8398e6f89bf5419d81ed5bed991da0cf, b052e89b63c24258872c832a6a0d4d4d; no voters:
I20240503 08:00:23.940533 22610 raft_consensus.cc:2781] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:23.940598 22610 raft_consensus.cc:686] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:23.940734 22610 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:23.942902 12671 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:23.943748 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cee7c94dd0941d99d9dd6ceb39fc190" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:23.943868 12946 raft_consensus.cc:3037] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:23.944861 12946 raft_consensus.cc:2454] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:23.961968 14789 raft_consensus.cc:1203] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:23.964807 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.006004 14789 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
W20240503 08:00:24.008294 14724 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): Couldn't send request to peer 6c358da9faef48c4adb12cce628aeb38. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.008301 13077 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba"
dest_uuid: "39827620daed48cb8f137442e1f1da76"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46380
I20240503 08:00:24.008442 13077 raft_consensus.cc:595] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:24.009037 13206 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 08:00:24.009202 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ac9fbe5dfd4a4e89912d7e947eca45. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.009549 22225 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.053117 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:24.058270 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.061178 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.063120 13057 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:00:24.063601 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:24.071652 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.083060 14769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.107326 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.110206 22380 raft_consensus.cc:979] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: : Instructing follower af712db2f85249adbe718e8ba0549f84 to start an election
I20240503 08:00:24.110286 22380 raft_consensus.cc:1067] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Signalling peer af712db2f85249adbe718e8ba0549f84 to start an election
I20240503 08:00:24.111766 13206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
from {username='slave'} at 127.0.99.131:47613
I20240503 08:00:24.111912 13206 raft_consensus.cc:482] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:24.111963 13206 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.112990 13206 raft_consensus.cc:504] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:24.113402 13206 leader_election.cc:288] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 2 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:24.113688 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:24.113739 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0d84a47cc34af9bdd6b955feb2c3ba" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8398e6f89bf5419d81ed5bed991da0cf"
I20240503 08:00:24.113795 13077 raft_consensus.cc:3032] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:24.113829 13077 raft_consensus.cc:729] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:00:24.113857 14657 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.113970 13077 consensus_queue.cc:260] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:24.114111 13077 raft_consensus.cc:3037] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.115097 13077 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 2.
I20240503 08:00:24.115453 13141 leader_election.cc:302] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:00:24.116427 22739 raft_consensus.cc:2781] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:24.116508 22739 raft_consensus.cc:686] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 LEADER]: Becoming Leader. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:24.116657 22739 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:24.117401 14657 raft_consensus.cc:2454] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af712db2f85249adbe718e8ba0549f84 in term 2.
I20240503 08:00:24.118517 12665 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 reported cstate change: term changed from 1 to 2, leader changed from 39827620daed48cb8f137442e1f1da76 (127.0.99.131) to af712db2f85249adbe718e8ba0549f84 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:24.185541 14919 raft_consensus.cc:1203] T 9e7f229898c24572815177628720521f P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:24.191876 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.196556 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:24.204574 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.205377 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.217023 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.217463 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.228833 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.233673 14769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.245608 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.250942 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:24.253305 14769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.262140 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.276192 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.287755 13206 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:24.288156 22741 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:24.293941 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.299227 12815 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:24.300374 19382 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:24.309305 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.310513 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.312337 12924 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:24.313911 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.318740 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.327387 22927 ts_tablet_manager.cc:902] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:24.328250 22927 tablet_copy_client.cc:308] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:24.328832 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 9c7bf40aa0614923a19feed8bab430fa from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:41249)
I20240503 08:00:24.328948 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 9c7bf40aa0614923a19feed8bab430fa from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:41249: session id = 0cf63591cfc244028e1fc368c2bca079-9c7bf40aa0614923a19feed8bab430fa
I20240503 08:00:24.329988 14808 tablet_copy_source_session.cc:205] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.331001 22927 ts_tablet_manager.cc:1875] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:24.334148 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.340058 14769 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.344691 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.368397 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.370121 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.371083 22927 ts_tablet_manager.cc:1888] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:00:24.371366 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7bf40aa0614923a19feed8bab430fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.373415 22927 tablet_copy_client.cc:783] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.373611 22927 tablet_copy_client.cc:647] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.387749 22927 tablet_copy_client.cc:523] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.389550 22927 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:24.395236 14789 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:24.396400 14526 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:24.396828 22225 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.405704 22225 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.407594 14526 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.409214 12672 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.412926 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:24.413259 14789 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:24.414662 14526 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:24.415014 22609 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.416340 22610 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
W20240503 08:00:24.417196 14725 consensus_peers.cc:485] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c630e5cf7c0419cbf5d13413d22d317. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.418028 14526 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:24.419006 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7c630e5cf7c0419cbf5d13413d22d317 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:24.420595 12672 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:24.425839 22971 tablet_replica.cc:330] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:24.426132 22971 raft_consensus.cc:2229] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.426326 12946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48e4f2a993164fb6b1df674311698006"
dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33780
I20240503 08:00:24.426443 12946 raft_consensus.cc:595] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:24.426617 22971 raft_consensus.cc:2258] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.427359 22971 ts_tablet_manager.cc:1875] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:24.430014 22971 ts_tablet_manager.cc:1888] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 08:00:24.430088 22971 log.cc:1196] T 7c630e5cf7c0419cbf5d13413d22d317 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:24.430544 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 7c630e5cf7c0419cbf5d13413d22d317 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:00:24.432735 22927 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.433074 22927 tablet_bootstrap.cc:492] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:24.433230 22927 ts_tablet_manager.cc:1367] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.002s
I20240503 08:00:24.433656 22927 raft_consensus.cc:348] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:24.433782 22927 raft_consensus.cc:729] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:24.433925 22927 consensus_queue.cc:260] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:24.434798 22927 ts_tablet_manager.cc:1398] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:24.435303 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-9c7bf40aa0614923a19feed8bab430fa received from {username='slave'} at 127.0.99.130:41249
I20240503 08:00:24.435386 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-9c7bf40aa0614923a19feed8bab430fa on tablet 9c7bf40aa0614923a19feed8bab430fa with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:24.444193 12946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48e4f2a993164fb6b1df674311698006"
dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33786
I20240503 08:00:24.444319 12946 raft_consensus.cc:595] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:24.480885 12815 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.481317 22609 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.503979 14526 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.504624 22609 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.509482 14657 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.512198 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.512831 22511 raft_consensus.cc:979] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: : Instructing follower b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:24.512907 22511 raft_consensus.cc:1067] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Signalling peer b052e89b63c24258872c832a6a0d4d4d to start an election
I20240503 08:00:24.513715 22610 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.514858 22225 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.519860 14789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48e4f2a993164fb6b1df674311698006"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
from {username='slave'} at 127.0.99.130:41249
I20240503 08:00:24.519994 14789 raft_consensus.cc:482] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:24.520051 14789 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.521092 14789 raft_consensus.cc:504] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:24.521445 14789 leader_election.cc:288] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Requested vote from peers 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937), eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:24.522205 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "eae9f53215a34bc88abe7b3c1c223319"
I20240503 08:00:24.522341 14919 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.523274 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4f2a993164fb6b1df674311698006" candidate_uuid: "b052e89b63c24258872c832a6a0d4d4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:00:24.523389 12944 raft_consensus.cc:3032] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:24.523429 12944 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Running, Role: LEADER
I20240503 08:00:24.523564 12944 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:24.523746 12944 raft_consensus.cc:3037] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.524704 12944 raft_consensus.cc:2454] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:24.525774 14919 raft_consensus.cc:2454] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b052e89b63c24258872c832a6a0d4d4d in term 2.
I20240503 08:00:24.526116 14722 leader_election.cc:302] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b052e89b63c24258872c832a6a0d4d4d, eae9f53215a34bc88abe7b3c1c223319; no voters:
I20240503 08:00:24.526322 22610 raft_consensus.cc:2781] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:24.526391 22610 raft_consensus.cc:686] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Leader. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:00:24.526516 22610 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:24.535890 12665 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: term changed from 1 to 2, leader changed from 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) to b052e89b63c24258872c832a6a0d4d4d (127.0.99.135). New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:00:24.542236 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.549737 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.553344 23120 ts_tablet_manager.cc:902] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:24.556474 23120 tablet_copy_client.cc:308] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:24.556970 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer 6c358da9faef48c4adb12cce628aeb38 ({username='slave'} at 127.0.99.129:57591)
I20240503 08:00:24.557060 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer 6c358da9faef48c4adb12cce628aeb38 at {username='slave'} at 127.0.99.129:57591: session id = 6c358da9faef48c4adb12cce628aeb38-72ac9fbe5dfd4a4e89912d7e947eca45
I20240503 08:00:24.557994 14808 tablet_copy_source_session.cc:205] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.563215 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.563318 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.565958 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.567698 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.580612 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.594961 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.595396 23120 ts_tablet_manager.cc:1875] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:24.597714 23120 ts_tablet_manager.cc:1888] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 08:00:24.597884 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.599236 23120 tablet_copy_client.cc:783] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.599390 23120 tablet_copy_client.cc:647] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.602483 23134 ts_tablet_manager.cc:902] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:24.602967 23134 tablet_copy_client.cc:308] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:24.604808 13077 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:24.605639 22739 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:24.614295 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.614876 23120 tablet_copy_client.cc:523] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.615486 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.616356 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.617452 23120 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:00:24.620738 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.639722 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.639860 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.644856 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.649313 23120 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:24.649677 23120 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap complete.
I20240503 08:00:24.649855 23120 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.000s
I20240503 08:00:24.650272 23120 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:24.650492 23120 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:00:24.650580 23120 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:24.651450 23120 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:00:24.651983 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 6c358da9faef48c4adb12cce628aeb38-72ac9fbe5dfd4a4e89912d7e947eca45 received from {username='slave'} at 127.0.99.129:57591
I20240503 08:00:24.652057 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 6c358da9faef48c4adb12cce628aeb38-72ac9fbe5dfd4a4e89912d7e947eca45 on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with peer 6c358da9faef48c4adb12cce628aeb38
I20240503 08:00:24.658528 14658 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:24.658995 21327 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:24.662232 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer eae9f53215a34bc88abe7b3c1c223319 ({username='slave'} at 127.0.99.136:60281)
I20240503 08:00:24.662330 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 from peer eae9f53215a34bc88abe7b3c1c223319 at {username='slave'} at 127.0.99.136:60281: session id = eae9f53215a34bc88abe7b3c1c223319-72ac9fbe5dfd4a4e89912d7e947eca45
I20240503 08:00:24.663129 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.663345 14808 tablet_copy_source_session.cc:205] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.664227 23134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ac9fbe5dfd4a4e89912d7e947eca45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.666399 23134 tablet_copy_client.cc:783] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.666534 23134 tablet_copy_client.cc:647] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.667889 23134 tablet_copy_client.cc:523] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.671841 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.671918 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.679565 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.685278 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.691690 23134 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap starting.
I20240503 08:00:24.699993 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.706670 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.711292 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.739267 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.745805 23134 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:24.750450 23134 tablet_bootstrap.cc:492] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Bootstrap complete.
I20240503 08:00:24.750676 23134 ts_tablet_manager.cc:1367] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.005s
I20240503 08:00:24.751180 23134 raft_consensus.cc:348] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:24.751431 23134 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9f53215a34bc88abe7b3c1c223319, State: Initialized, Role: FOLLOWER
I20240503 08:00:24.751533 23134 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:00:24.751930 23134 ts_tablet_manager.cc:1398] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:24.752704 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session eae9f53215a34bc88abe7b3c1c223319-72ac9fbe5dfd4a4e89912d7e947eca45 received from {username='slave'} at 127.0.99.136:60281
I20240503 08:00:24.752789 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session eae9f53215a34bc88abe7b3c1c223319-72ac9fbe5dfd4a4e89912d7e947eca45 on tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with peer eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:24.769286 12946 raft_consensus.cc:1203] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240503 08:00:24.797080 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.799171 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:24.799686 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.802238 22927 ts_tablet_manager.cc:902] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:24.805215 22927 tablet_copy_client.cc:308] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:24.805727 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:41249)
I20240503 08:00:24.805821 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:41249: session id = 0cf63591cfc244028e1fc368c2bca079-7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:24.806842 14808 tablet_copy_source_session.cc:205] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.807889 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.810459 22927 tablet_copy_client.cc:783] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.811857 22927 tablet_copy_client.cc:647] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.819710 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.820662 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:24.822016 12815 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:24.826301 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:24.827558 14526 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:24.827927 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.827965 21733 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.829368 22927 tablet_copy_client.cc:523] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.831046 22927 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:24.832489 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.868517 21733 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.872267 12665 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.874488 14526 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:24.878486 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44532
I20240503 08:00:24.878803 12815 consensus_queue.cc:237] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
W20240503 08:00:24.880723 12748 consensus_peers.cc:485] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 -> Peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): Couldn't send request to peer af712db2f85249adbe718e8ba0549f84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcdde0a31f7d4fd0bf20c894f215e3b6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.881394 14526 raft_consensus.cc:1261] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.881774 21733 consensus_queue.cc:1035] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.883718 22535 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:24.884227 14526 raft_consensus.cc:2932] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:24.886181 12665 catalog_manager.cc:5484] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:24.886561 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:24.888741 23242 tablet_replica.cc:330] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:00:24.888882 23242 raft_consensus.cc:2229] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.889005 23242 raft_consensus.cc:2258] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.889451 23242 ts_tablet_manager.cc:1875] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:24.896302 23242 ts_tablet_manager.cc:1888] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:24.896379 23242 log.cc:1196] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcdde0a31f7d4fd0bf20c894f215e3b6
I20240503 08:00:24.896910 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet dcdde0a31f7d4fd0bf20c894f215e3b6 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:00:24.904337 22927 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.904597 22927 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:24.904800 22927 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.074s user 0.026s sys 0.000s
I20240503 08:00:24.905272 22927 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:24.905383 22927 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:24.905524 22927 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:24.905858 22927 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:24.906493 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-7c630e5cf7c0419cbf5d13413d22d317 received from {username='slave'} at 127.0.99.130:41249
I20240503 08:00:24.906589 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-7c630e5cf7c0419cbf5d13413d22d317 on tablet 7c630e5cf7c0419cbf5d13413d22d317 with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:24.919811 12944 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:24.921309 14920 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.921655 14526 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:24.922426 23096 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.922746 22511 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.936640 22511 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:24.939312 14526 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:24.941829 12671 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 6 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:24.943859 14920 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:24.946794 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet c917c5849f54411eb55d69ab33456bf4 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:24.947217 23297 tablet_replica.cc:330] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:24.947388 23297 raft_consensus.cc:2229] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.947535 23297 raft_consensus.cc:2258] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.948179 23297 ts_tablet_manager.cc:1875] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:24.950572 23297 ts_tablet_manager.cc:1888] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:00:24.950750 23297 log.cc:1196] T c917c5849f54411eb55d69ab33456bf4 P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c917c5849f54411eb55d69ab33456bf4
I20240503 08:00:24.952306 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet c917c5849f54411eb55d69ab33456bf4 (table rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]) successfully deleted
I20240503 08:00:25.001457 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.005479 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:25.012872 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.022778 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.025002 14920 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.025324 12944 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.025457 22609 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.025938 22609 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.032420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.033335 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.048499 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.049612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.054333 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.073761 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:25.074182 12815 raft_consensus.cc:1203] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 08:00:25.075436 23081 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:25.076301 12815 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:25.076923 23081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54202 (request call id 0) took 1071 ms. Trace:
I20240503 08:00:25.076994 23081 rpcz_store.cc:276] 0503 08:00:24.005460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:00:24.005498 (+ 38us) service_pool.cc:225] Handling call
0503 08:00:24.006631 (+ 1133us) log.cc:842] Serialized 482 byte log entry
0503 08:00:25.076909 (+1070278us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1625,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":256}
I20240503 08:00:25.079643 12669 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:25.082805 13208 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:25.087915 14920 raft_consensus.cc:1203] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 08:00:25.089672 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.092578 14920 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:00:25.103775 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.118577 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.119076 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:25.129985 23366 log.cc:924] Time spent T 48e4f2a993164fb6b1df674311698006 P eae9f53215a34bc88abe7b3c1c223319: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20240503 08:00:25.131855 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.132416 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.168470 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.172715 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.177481 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.178422 14507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.178994 13188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:25.190639 12944 raft_consensus.cc:1203] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 08:00:25.202069 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.204974 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.207115 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.235525 13208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1"
dest_uuid: "af712db2f85249adbe718e8ba0549f84"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51938
I20240503 08:00:25.235688 13208 raft_consensus.cc:595] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:25.240609 21327 raft_consensus.cc:979] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84: : Instructing follower 82cd33f2bae4419b946095f8eb93e1e4 to start an election
I20240503 08:00:25.240741 23440 raft_consensus.cc:1067] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Signalling peer 82cd33f2bae4419b946095f8eb93e1e4 to start an election
I20240503 08:00:25.241333 14526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1"
dest_uuid: "82cd33f2bae4419b946095f8eb93e1e4"
from {username='slave'} at 127.0.99.132:48927
I20240503 08:00:25.241492 14526 raft_consensus.cc:482] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:25.241567 14526 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:25.242641 14526 raft_consensus.cc:504] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:25.243116 14526 leader_election.cc:288] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [CANDIDATE]: Term 2 election: Requested vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323)
I20240503 08:00:25.243958 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "82cd33f2bae4419b946095f8eb93e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "eae9f53215a34bc88abe7b3c1c223319"
I20240503 08:00:25.244086 14919 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:25.245114 14919 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd33f2bae4419b946095f8eb93e1e4 in term 2.
I20240503 08:00:25.245474 13077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "82cd33f2bae4419b946095f8eb93e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "39827620daed48cb8f137442e1f1da76"
I20240503 08:00:25.245591 13077 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:25.246544 13077 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd33f2bae4419b946095f8eb93e1e4 in term 2.
I20240503 08:00:25.247416 14460 leader_election.cc:302] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 39827620daed48cb8f137442e1f1da76, 82cd33f2bae4419b946095f8eb93e1e4, eae9f53215a34bc88abe7b3c1c223319; no voters:
I20240503 08:00:25.247839 18526 raft_consensus.cc:2781] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:25.247861 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "82cd33f2bae4419b946095f8eb93e1e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:00:25.247983 18526 raft_consensus.cc:686] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Running, Role: LEADER
I20240503 08:00:25.247992 13208 raft_consensus.cc:3032] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:25.248036 13208 raft_consensus.cc:729] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Running, Role: LEADER
I20240503 08:00:25.248128 18526 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:25.248178 13208 consensus_queue.cc:260] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:25.248381 13208 raft_consensus.cc:3037] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:25.249250 13208 raft_consensus.cc:2454] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd33f2bae4419b946095f8eb93e1e4 in term 2.
I20240503 08:00:25.257087 12671 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: term changed from 1 to 2, leader changed from af712db2f85249adbe718e8ba0549f84 (127.0.99.132) to 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:25.282116 14788 consensus_queue.cc:237] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:25.283471 12944 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.283500 12815 raft_consensus.cc:1261] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.283874 22609 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.284225 22609 consensus_queue.cc:1035] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.285646 22610 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.286444 12815 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.289623 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.289649 12664 catalog_manager.cc:5484] T 9c7bf40aa0614923a19feed8bab430fa P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.293969 12944 raft_consensus.cc:2932] T 9c7bf40aa0614923a19feed8bab430fa P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.295871 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7bf40aa0614923a19feed8bab430fa with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38966
I20240503 08:00:25.296092 23297 tablet_replica.cc:330] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:25.296233 23297 raft_consensus.cc:2229] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:25.296357 23297 raft_consensus.cc:2258] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:25.296976 23297 ts_tablet_manager.cc:1875] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:25.300215 23297 ts_tablet_manager.cc:1888] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 08:00:25.300318 23297 log.cc:1196] T 9c7bf40aa0614923a19feed8bab430fa P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c7bf40aa0614923a19feed8bab430fa
I20240503 08:00:25.301188 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 9c7bf40aa0614923a19feed8bab430fa (table rebalance_test_table_0 [id=d0a7085823ec47c59ba5179f5a70d9e8]) successfully deleted
I20240503 08:00:25.304288 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.308530 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.309031 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.323818 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:25.326301 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:25.328719 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.329469 14788 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:25.330816 14526 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:00:25.331252 22609 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:25.331710 12944 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:00:25.332059 22610 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:25.338191 14722 consensus_peers.cc:485] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d -> Peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): Couldn't send request to peer af712db2f85249adbe718e8ba0549f84. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.347461 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.366680 22610 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:25.368034 12944 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:25.370743 12673 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 10 to 11, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:25.371534 14526 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:25.379388 23502 ts_tablet_manager.cc:902] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:25.380077 23502 tablet_copy_client.cc:308] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:25.380652 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet dcdde0a31f7d4fd0bf20c894f215e3b6 from peer af712db2f85249adbe718e8ba0549f84 ({username='slave'} at 127.0.99.132:60853)
I20240503 08:00:25.380753 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet dcdde0a31f7d4fd0bf20c894f215e3b6 from peer af712db2f85249adbe718e8ba0549f84 at {username='slave'} at 127.0.99.132:60853: session id = af712db2f85249adbe718e8ba0549f84-dcdde0a31f7d4fd0bf20c894f215e3b6
I20240503 08:00:25.381656 12835 tablet_copy_source_session.cc:205] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:25.383582 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdde0a31f7d4fd0bf20c894f215e3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:25.386282 23502 tablet_copy_client.cc:783] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:25.386462 23502 tablet_copy_client.cc:647] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:25.387853 23502 tablet_copy_client.cc:523] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:25.390266 23502 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:00:25.420218 23502 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:25.420495 23502 tablet_bootstrap.cc:492] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Bootstrap complete.
I20240503 08:00:25.420656 23502 ts_tablet_manager.cc:1367] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240503 08:00:25.421103 23502 raft_consensus.cc:348] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:25.421223 23502 raft_consensus.cc:729] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:25.424614 23502 consensus_queue.cc:260] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:25.426009 13208 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:25.427515 14657 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.428484 13077 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.428671 23579 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:25.430315 13141 consensus_peers.cc:485] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e0d84a47cc34af9bdd6b955feb2c3ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.431694 23440 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:25.433241 23502 ts_tablet_manager.cc:1398] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:00:25.433919 23440 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.435325 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session af712db2f85249adbe718e8ba0549f84-dcdde0a31f7d4fd0bf20c894f215e3b6 received from {username='slave'} at 127.0.99.132:60853
I20240503 08:00:25.435410 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session af712db2f85249adbe718e8ba0549f84-dcdde0a31f7d4fd0bf20c894f215e3b6 on tablet dcdde0a31f7d4fd0bf20c894f215e3b6 with peer af712db2f85249adbe718e8ba0549f84
I20240503 08:00:25.436618 14657 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.437675 12673 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 5 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.443719 13077 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.465351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.483894 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.495301 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:25.499159 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.503661 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.519827 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.521353 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:25.523953 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.525115 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:25.538336 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:25.539466 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.542347 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.543018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.545433 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.569113 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.574338 12815 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
W20240503 08:00:25.576495 12751 consensus_peers.cc:485] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.580319 14788 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.580438 14526 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.580843 21733 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.581166 21733 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.587158 21733 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.590543 14526 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.590770 12668 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.593992 14788 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.599767 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.658244 13208 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:25.660938 14788 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:25.661538 13141 consensus_peers.cc:485] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c84f7a57e0945d9a25a70798f054bd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.661931 14658 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.662317 23579 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.663920 23579 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.673592 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.673950 23581 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.674312 14788 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.676156 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:25.678206 14658 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.678337 12668 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } }
I20240503 08:00:25.680610 13208 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.682965 23442 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:25.692965 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.702899 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.703931 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:25.724668 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.727123 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.761655 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.769466 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:25.781678 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:25.782415 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.782661 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:25.783088 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.785151 13208 raft_consensus.cc:1203] T dcdde0a31f7d4fd0bf20c894f215e3b6 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:25.797819 14658 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:25.799173 12815 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.799715 23749 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:25.800514 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.802659 23442 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:25.804191 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.806192 14919 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P eae9f53215a34bc88abe7b3c1c223319 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:25.808722 23812 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:25.811450 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:25.812566 14900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.814816 23502 ts_tablet_manager.cc:902] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:25.815339 23502 tablet_copy_client.cc:308] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:25.815816 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer af712db2f85249adbe718e8ba0549f84 ({username='slave'} at 127.0.99.132:52351)
I20240503 08:00:25.815920 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 7c630e5cf7c0419cbf5d13413d22d317 from peer af712db2f85249adbe718e8ba0549f84 at {username='slave'} at 127.0.99.132:52351: session id = af712db2f85249adbe718e8ba0549f84-7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:25.816794 14808 tablet_copy_source_session.cc:205] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:25.817639 23502 ts_tablet_manager.cc:1875] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:25.819604 23502 ts_tablet_manager.cc:1888] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 08:00:25.819730 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c630e5cf7c0419cbf5d13413d22d317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:25.820518 23502 tablet_copy_client.cc:783] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:25.820606 23502 tablet_copy_client.cc:647] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:25.829432 23502 tablet_copy_client.cc:523] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:25.830296 23834 ts_tablet_manager.cc:902] T 0e0d84a47cc34af9bdd6b955feb2c3ba P eae9f53215a34bc88abe7b3c1c223319: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
F20240503 08:00:25.830466 23834 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-047aba81fad543ce8b562a184746b163.queue_time_us, tablet-download-pool-047aba81fad543ce8b562a184746b163.queue_time_us) (tablet-download-pool-047aba81fad543ce8b562a184746b163.run_cpu_time_us, tablet-download-pool-047aba81fad543ce8b562a184746b163.run_cpu_time_us) (tablet-download-pool-047aba81fad543ce8b562a184746b163.run_wall_time_us, tablet-download-pool-047aba81fad543ce8b562a184746b163.run_wall_time_us) (tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.queue_time_us, tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.queue_time_us) (tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.run_cpu_time_us, tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.run_cpu_time_us) (tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.run_wall_time_us, tablet-download-pool-076781af2ac14ddc87d31e3ec77db08f.run_wall_time_us) (tablet-download-pool-0e0d84a47cc34af9bdd6b955feb2c3ba.queue_time_us, tablet-download-pool-0e0d84a47cc34af9bdd6b955feb2c3ba.queue_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.queue_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.queue_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_cpu_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_cpu_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_wall_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_wall_time_us) (tablet-download-pool-48e4f2a993164fb6b1df674311698006.queue_time_us, tablet-download-pool-48e4f2a993164fb6b1df674311698006.queue_time_us) (tablet-download-pool-48e4f2a993164fb6b1df674311698006.run_cpu_time_us, tablet-download-pool-48e4f2a993164fb6b1df674311698006.run_cpu_time_us) (tablet-download-pool-48e4f2a993164fb6b1df674311698006.run_wall_time_us, tablet-download-pool-48e4f2a993164fb6b1df674311698006.run_wall_time_us) (tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.queue_time_us, tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.queue_time_us) (tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.run_cpu_time_us, tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.run_cpu_time_us) (tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.run_wall_time_us, tablet-download-pool-672fc9981d074f7e8f2324ba0b06be57.run_wall_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.queue_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.queue_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_cpu_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_cpu_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_wall_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_wall_time_us) (tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.queue_time_us, tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.queue_time_us) (tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.run_cpu_time_us, tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.run_cpu_time_us) (tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.run_wall_time_us, tablet-download-pool-72ac9fbe5dfd4a4e89912d7e947eca45.run_wall_time_us) (tablet-download-pool-7d61aa445a50417b82916019c2a4298c.queue_time_us, tablet-download-pool-7d61aa445a50417b82916019c2a4298c.queue_time_us) (tablet-download-pool-7d61aa445a50417b82916019c2a4298c.run_cpu_time_us, tablet-download-pool-7d61aa445a50417b82916019c2a4298c.run_cpu_time_us) (tablet-download-pool-7d61aa445a50417b82916019c2a4298c.run_wall_time_us, tablet-download-pool-7d61aa445a50417b82916019c2a4298c.run_wall_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.queue_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.queue_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.run_cpu_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.run_cpu_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.run_wall_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.run_wall_time_us) (tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.queue_time_us, tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.queue_time_us) (tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.run_cpu_time_us, tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.run_cpu_time_us) (tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.run_wall_time_us, tablet-download-pool-a45748a98c3447d3aa2e8de94981a745.run_wall_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.queue_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.queue_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_cpu_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_cpu_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_wall_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_wall_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.queue_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.queue_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_cpu_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_cpu_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_wall_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_wall_time_us) (tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.queue_time_us, tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.queue_time_us) (tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.run_cpu_time_us, tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.run_cpu_time_us) (tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.run_wall_time_us, tablet-download-pool-d376ac1ea05c482ca8f03ddf0f5fcd47.run_wall_time_us) (tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.queue_time_us, tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.queue_time_us) (tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.run_cpu_time_us, tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.run_cpu_time_us) (tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.run_wall_time_us, tablet-download-pool-d8994c38d0f640e9bd5b5c322c603b9f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 08:00:25.830847 23502 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
*** Check failure stack trace: ***
@ 0x7f491696cdcd google::LogMessage::Fail() at ??:0
@ 0x7f4916970b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f491696c7cc google::LogMessage::Flush() at ??:0
@ 0x7f491696df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4917dd6044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4917dbaf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4917db8efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f491a96b77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f491a974a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:00:25.856536 23749 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.858678 12815 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.860030 12665 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 5 to 7, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.863245 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3e4066cd14aa18619310a4490246b with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38966
@ 0x7f491a9f576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:00:25.866546 23842 tablet_replica.cc:330] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: stopping tablet replica
I20240503 08:00:25.866716 23842 raft_consensus.cc:2229] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:25.866833 23842 raft_consensus.cc:2258] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:25.867327 23842 ts_tablet_manager.cc:1875] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:25.868906 14658 consensus_queue.cc:237] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
@ 0x7f491aa094f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240503 08:00:25.870299 14594 consensus_peers.cc:485] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02d3e4066cd14aa18619310a4490246b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.872083 12813 raft_consensus.cc:1261] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.873327 23749 consensus_queue.cc:1035] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.874413 23842 ts_tablet_manager.cc:1888] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:25.874476 23842 log.cc:1196] T 02d3e4066cd14aa18619310a4490246b P af712db2f85249adbe718e8ba0549f84: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/02d3e4066cd14aa18619310a4490246b
I20240503 08:00:25.875059 12657 catalog_manager.cc:4830] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): tablet 02d3e4066cd14aa18619310a4490246b (table rebalance_test_table_3 [id=4aa669a3d80c4dc492176981a1978274]) successfully deleted
I20240503 08:00:25.875273 23749 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
@ 0x7f491a9f4226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 08:00:25.877009 12658 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 02d3e4066cd14aa18619310a4490246b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:00:25.877717 12813 raft_consensus.cc:2932] T 02d3e4066cd14aa18619310a4490246b P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:25.878072 12665 catalog_manager.cc:5484] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 7 to 8, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
@ 0x7f491aa03c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ea8f2c675c std::function<>::operator()() at ??:0
@ 0x7f4917dbde06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4917dbe6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4917dc003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ea8f2c675c std::function<>::operator()() at ??:0
@ 0x7f4917dad807 kudu::Thread::SuperviseThread() at ??:0
I20240503 08:00:25.895139 23502 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f49185eb6db start_thread at ??:0
I20240503 08:00:25.895416 23502 tablet_bootstrap.cc:492] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Bootstrap complete.
I20240503 08:00:25.895596 23502 ts_tablet_manager.cc:1367] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.065s user 0.026s sys 0.000s
@ 0x7f4915b2371f clone at ??:0
I20240503 08:00:25.901660 23502 raft_consensus.cc:348] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:25.901791 23502 raft_consensus.cc:729] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:00:25.901917 23502 consensus_queue.cc:260] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:25.902894 23502 ts_tablet_manager.cc:1398] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:25.907502 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session af712db2f85249adbe718e8ba0549f84-7c630e5cf7c0419cbf5d13413d22d317 received from {username='slave'} at 127.0.99.132:52351
I20240503 08:00:25.907683 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session af712db2f85249adbe718e8ba0549f84-7c630e5cf7c0419cbf5d13413d22d317 on tablet 7c630e5cf7c0419cbf5d13413d22d317 with peer af712db2f85249adbe718e8ba0549f84
I20240503 08:00:25.946894 14788 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:25.948506 12944 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:25.948596 14658 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:25.948884 23501 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.949499 23501 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.951431 14526 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:25.951820 23501 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.952627 14526 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
W20240503 08:00:25.954130 14463 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.956947 13076 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:25.957446 23812 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.963909 23501 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.966627 14658 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.967233 12946 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.969213 12667 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 7 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.973912 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with delete_type TABLET_DATA_TOMBSTONED (TS 6c358da9faef48c4adb12cce628aeb38 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52402
I20240503 08:00:25.976133 23923 tablet_replica.cc:330] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: stopping tablet replica
I20240503 08:00:25.976263 23923 raft_consensus.cc:2229] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:25.976358 23923 raft_consensus.cc:2258] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:25.976840 23923 ts_tablet_manager.cc:1875] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:25.981405 23923 ts_tablet_manager.cc:1888] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 08:00:25.981469 23923 log.cc:1196] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 6c358da9faef48c4adb12cce628aeb38: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:25.981967 12659 catalog_manager.cc:4830] TS 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
I20240503 08:00:25.982686 14527 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:25.995714 23953 ts_tablet_manager.cc:902] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:00:25.996213 23953 tablet_copy_client.cc:308] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:00:25.996665 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 389b40c2136d45ab86ccf49df8d1b71e from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:56231)
I20240503 08:00:25.996764 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 389b40c2136d45ab86ccf49df8d1b71e from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:56231: session id = 0cf63591cfc244028e1fc368c2bca079-389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:25.997696 12835 tablet_copy_source_session.cc:205] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:26.000777 23953 ts_tablet_manager.cc:1875] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:26.017902 23953 ts_tablet_manager.cc:1888] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:00:26.018105 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b40c2136d45ab86ccf49df8d1b71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.022128 23953 tablet_copy_client.cc:783] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:26.022281 23953 tablet_copy_client.cc:647] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:26.023756 23953 tablet_copy_client.cc:523] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:26.025152 23953 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:26.029769 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.035346 14788 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:26.037137 12946 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:26.037578 23889 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:00:26.038442 14723 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e4f2a993164fb6b1df674311698006. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:26.046377 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.054154 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.059739 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.061077 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.095234 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.133459 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.142060 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.148025 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.172552 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.182468 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.194294 23953 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:26.194649 23953 tablet_bootstrap.cc:492] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:26.194839 23953 ts_tablet_manager.cc:1367] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.170s user 0.027s sys 0.001s
I20240503 08:00:26.195297 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.195374 23953 raft_consensus.cc:348] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:26.195493 23953 raft_consensus.cc:729] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.197211 23953 consensus_queue.cc:260] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:26.201094 23953 ts_tablet_manager.cc:1398] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:26.202970 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-389b40c2136d45ab86ccf49df8d1b71e received from {username='slave'} at 127.0.99.130:56231
I20240503 08:00:26.203066 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-389b40c2136d45ab86ccf49df8d1b71e on tablet 389b40c2136d45ab86ccf49df8d1b71e with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:26.214068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.217906 13208 raft_consensus.cc:1203] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240503 08:00:26.218348 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.231104 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.231207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.235754 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.241168 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.251993 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.260237 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:26.286381 12748 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.286507 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.286826 14460 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.286962 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.286988 12879 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.287102 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.287495 14591 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.287537 13010 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.287590 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.287638 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.288069 13141 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.288169 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.288209 14722 connection.cc:535] client connection to 127.0.99.136:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.288301 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:26.294281 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.295271 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.298000 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.308799 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.312539 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.312620 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:26.312650 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:26.318109 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.318189 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:26.329572 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:26.336429 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.336764 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.336840 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.336889 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.336926 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.336963 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.337000 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.368042 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:26.388068 23953 ts_tablet_manager.cc:902] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:00:26.388533 23953 tablet_copy_client.cc:308] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38311
I20240503 08:00:26.388954 14547 tablet_copy_service.cc:140] P 82cd33f2bae4419b946095f8eb93e1e4: Received BeginTabletCopySession request for tablet 70c5de226c7a4b349eef86a79d172851 from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:37347)
I20240503 08:00:26.389060 14547 tablet_copy_service.cc:161] P 82cd33f2bae4419b946095f8eb93e1e4: Beginning new tablet copy session on tablet 70c5de226c7a4b349eef86a79d172851 from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:37347: session id = 0cf63591cfc244028e1fc368c2bca079-70c5de226c7a4b349eef86a79d172851
I20240503 08:00:26.390058 14547 tablet_copy_source_session.cc:205] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:26.390776 23953 ts_tablet_manager.cc:1875] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:26.392997 23953 ts_tablet_manager.cc:1888] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:00:26.393177 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c5de226c7a4b349eef86a79d172851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.394165 23953 tablet_copy_client.cc:783] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:26.394304 23953 tablet_copy_client.cc:647] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:26.395632 23953 tablet_copy_client.cc:523] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:26.397070 23953 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:26.397408 24122 ts_tablet_manager.cc:902] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:00:26.397892 24122 tablet_copy_client.cc:308] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39483
I20240503 08:00:26.399230 14678 tablet_copy_service.cc:140] P 8398e6f89bf5419d81ed5bed991da0cf: Received BeginTabletCopySession request for tablet 02d3e4066cd14aa18619310a4490246b from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:52549)
I20240503 08:00:26.399340 14678 tablet_copy_service.cc:161] P 8398e6f89bf5419d81ed5bed991da0cf: Beginning new tablet copy session on tablet 02d3e4066cd14aa18619310a4490246b from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:52549: session id = 0cf63591cfc244028e1fc368c2bca079-02d3e4066cd14aa18619310a4490246b
I20240503 08:00:26.400357 14678 tablet_copy_source_session.cc:205] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:26.401139 24122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3e4066cd14aa18619310a4490246b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.405766 24122 tablet_copy_client.cc:783] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:26.405915 24122 tablet_copy_client.cc:647] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:26.408581 24122 tablet_copy_client.cc:523] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:26.409894 24122 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:26.423159 23953 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:26.423446 23953 tablet_bootstrap.cc:492] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:26.423591 23953 ts_tablet_manager.cc:1367] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.003s
I20240503 08:00:26.423955 23953 raft_consensus.cc:348] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:26.424201 23953 raft_consensus.cc:729] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.424312 23953 consensus_queue.cc:260] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:26.424593 23953 ts_tablet_manager.cc:1398] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:26.425086 14547 tablet_copy_service.cc:342] P 82cd33f2bae4419b946095f8eb93e1e4: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-70c5de226c7a4b349eef86a79d172851 received from {username='slave'} at 127.0.99.130:37347
I20240503 08:00:26.425186 14547 tablet_copy_service.cc:433] P 82cd33f2bae4419b946095f8eb93e1e4: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-70c5de226c7a4b349eef86a79d172851 on tablet 70c5de226c7a4b349eef86a79d172851 with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:26.443157 24122 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:26.443475 24122 tablet_bootstrap.cc:492] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:26.443671 24122 ts_tablet_manager.cc:1367] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.005s
I20240503 08:00:26.444151 24122 raft_consensus.cc:348] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:26.444283 24122 raft_consensus.cc:729] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.444444 24122 consensus_queue.cc:260] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:26.444746 24122 ts_tablet_manager.cc:1398] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:26.445571 14678 tablet_copy_service.cc:342] P 8398e6f89bf5419d81ed5bed991da0cf: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-02d3e4066cd14aa18619310a4490246b received from {username='slave'} at 127.0.99.130:52549
I20240503 08:00:26.445657 14678 tablet_copy_service.cc:433] P 8398e6f89bf5419d81ed5bed991da0cf: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-02d3e4066cd14aa18619310a4490246b on tablet 02d3e4066cd14aa18619310a4490246b with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:26.461983 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.464435 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.466491 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.469712 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.471201 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.472467 24175 ts_tablet_manager.cc:902] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:00:26.482275 24175 tablet_copy_client.cc:308] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:00:26.487591 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.496912 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.501550 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 48e4f2a993164fb6b1df674311698006 from peer 8398e6f89bf5419d81ed5bed991da0cf ({username='slave'} at 127.0.99.134:46403)
I20240503 08:00:26.501658 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 48e4f2a993164fb6b1df674311698006 from peer 8398e6f89bf5419d81ed5bed991da0cf at {username='slave'} at 127.0.99.134:46403: session id = 8398e6f89bf5419d81ed5bed991da0cf-48e4f2a993164fb6b1df674311698006
I20240503 08:00:26.502652 14808 tablet_copy_source_session.cc:205] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:26.503867 24175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.506309 24175 tablet_copy_client.cc:783] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:26.506497 24175 tablet_copy_client.cc:647] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:26.510668 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.525571 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.530344 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.533681 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.536222 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.539995 24175 tablet_copy_client.cc:523] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:26.541491 24175 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap starting.
I20240503 08:00:26.565429 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.573487 12946 raft_consensus.cc:1203] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:26.577387 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.586220 24175 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:26.586602 24175 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Bootstrap complete.
I20240503 08:00:26.586771 24175 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.000s
I20240503 08:00:26.587224 24175 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:00:26.587604 24175 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398e6f89bf5419d81ed5bed991da0cf, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.587710 24175 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:00:26.588045 24175 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:26.588804 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-48e4f2a993164fb6b1df674311698006 received from {username='slave'} at 127.0.99.134:46403
I20240503 08:00:26.588876 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 8398e6f89bf5419d81ed5bed991da0cf-48e4f2a993164fb6b1df674311698006 on tablet 48e4f2a993164fb6b1df674311698006 with peer 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:26.608343 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:26.677407 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.687068 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.693539 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.698477 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.707506 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.710134 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.715598 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.720233 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.725703 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.726079 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.733933 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.741014 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.748487 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.751593 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.764602 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.768230 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.774590 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.793565 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.795248 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.796772 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.797398 12946 raft_consensus.cc:1203] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 08:00:26.798369 23812 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:26.800576 13076 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:26.804524 12665 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 8 to 9, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:26.806061 12946 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:26.813190 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:26.927371 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:26.929453 12946 raft_consensus.cc:1203] T 02d3e4066cd14aa18619310a4490246b P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:26.942714 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:26.947367 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:26.951077 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:26.951390 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:26.958246 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:26.963451 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:26.983114 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.001791 14658 raft_consensus.cc:1203] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 08:00:27.002591 23889 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:27.005482 12946 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:27.005740 12668 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 7 to 9, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:27.008661 14658 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:00:27.031716 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.047133 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.047860 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.054126 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.054529 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.059727 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.064038 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.100437 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.105333 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.108884 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.117663 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.124373 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.129966 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.144232 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.153841 13208 consensus_queue.cc:237] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:27.156378 12813 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:27.157282 21327 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:00:27.157747 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:27.158901 13144 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:27.186367 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:27.211181 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.214452 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.216357 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.217187 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.226593 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.234341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.250713 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.308404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.308730 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.311575 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.316802 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.327606 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.327633 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.332417 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.384224 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.386567 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.389544 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.391538 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.396128 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.414371 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.414587 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:27.503679 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:27.510605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.528110 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.528895 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.530629 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.548214 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.553546 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.556219 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.563818 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.566692 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.567442 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.582885 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.593014 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.595736 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.624956 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.657393 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.664032 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.664672 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.665226 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.684197 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.692786 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:27.697630 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:27.705806 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.758968 24835 ts_tablet_manager.cc:902] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:00:27.759506 24835 tablet_copy_client.cc:308] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37969
I20240503 08:00:27.760028 13228 tablet_copy_service.cc:140] P af712db2f85249adbe718e8ba0549f84: Received BeginTabletCopySession request for tablet d376ac1ea05c482ca8f03ddf0f5fcd47 from peer 0cf63591cfc244028e1fc368c2bca079 ({username='slave'} at 127.0.99.130:56689)
I20240503 08:00:27.760183 13228 tablet_copy_service.cc:161] P af712db2f85249adbe718e8ba0549f84: Beginning new tablet copy session on tablet d376ac1ea05c482ca8f03ddf0f5fcd47 from peer 0cf63591cfc244028e1fc368c2bca079 at {username='slave'} at 127.0.99.130:56689: session id = 0cf63591cfc244028e1fc368c2bca079-d376ac1ea05c482ca8f03ddf0f5fcd47
I20240503 08:00:27.760929 13228 tablet_copy_source_session.cc:205] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:27.761998 24835 ts_tablet_manager.cc:1875] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:27.764518 24835 ts_tablet_manager.cc:1888] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 08:00:27.765849 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376ac1ea05c482ca8f03ddf0f5fcd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.767194 24835 tablet_copy_client.cc:783] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:27.767338 24835 tablet_copy_client.cc:647] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:27.768421 24835 tablet_copy_client.cc:523] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:27.769922 24835 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap starting.
I20240503 08:00:27.779155 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.784123 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.798413 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.811621 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.811707 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.815227 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.816030 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.827924 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.845170 24835 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:27.845477 24835 tablet_bootstrap.cc:492] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Bootstrap complete.
I20240503 08:00:27.845661 24835 ts_tablet_manager.cc:1367] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Time spent bootstrapping tablet: real 0.076s user 0.023s sys 0.000s
I20240503 08:00:27.846101 24835 raft_consensus.cc:348] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } }
I20240503 08:00:27.848598 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.848960 24835 raft_consensus.cc:729] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf63591cfc244028e1fc368c2bca079, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.849090 24835 consensus_queue.cc:260] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:27.849411 24835 ts_tablet_manager.cc:1398] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:27.851850 13228 tablet_copy_service.cc:342] P af712db2f85249adbe718e8ba0549f84: Request end of tablet copy session 0cf63591cfc244028e1fc368c2bca079-d376ac1ea05c482ca8f03ddf0f5fcd47 received from {username='slave'} at 127.0.99.130:56689
I20240503 08:00:27.851929 13228 tablet_copy_service.cc:433] P af712db2f85249adbe718e8ba0549f84: ending tablet copy session 0cf63591cfc244028e1fc368c2bca079-d376ac1ea05c482ca8f03ddf0f5fcd47 on tablet d376ac1ea05c482ca8f03ddf0f5fcd47 with peer 0cf63591cfc244028e1fc368c2bca079
I20240503 08:00:27.874954 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.887472 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:27.897570 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.899276 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.910341 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:27.911525 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.924023 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:27.929646 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:27.946975 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:27.948130 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:27.950032 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:27.955432 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:27.965436 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.035529 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.038561 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.043059 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.056720 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.058859 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.061929 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.085047 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.107689 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.119359 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.120658 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.126259 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.133975 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.135651 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.157189 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.194764 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:28.215515 12946 raft_consensus.cc:1203] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:28.216431 24513 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.217983 24513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52178 (request call id 0) took 1064 ms. Trace:
I20240503 08:00:28.218062 24513 rpcz_store.cc:276] 0503 08:00:27.153419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:00:27.153466 (+ 47us) service_pool.cc:225] Handling call
0503 08:00:27.154432 (+ 966us) log.cc:842] Serialized 482 byte log entry
0503 08:00:28.217967 (+1063535us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1089,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":256,"thread_start_us":183,"threads_started":3,"wal-append.queue_time_us":8624,"wal-append.run_cpu_time_us":278,"wal-append.run_wall_time_us":1000332}
I20240503 08:00:28.223045 12946 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.226089 12667 catalog_manager.cc:5484] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.229074 12813 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "eae9f53215a34bc88abe7b3c1c223319" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35323 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.229413 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.239907 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.241686 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.242590 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.249087 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.250494 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.264261 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:28.280334 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.323041 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.331600 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.333711 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.339128 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.344267 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:28.345677 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.352658 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.367118 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.384552 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:28.387462 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.404361 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.408176 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.408665 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.413710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.417774 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.419603 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.425117 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.437357 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:28.437922 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.451885 14788 consensus_queue.cc:237] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:28.453696 12946 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:28.454094 24444 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:28.454762 14527 raft_consensus.cc:1261] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:28.455144 25178 consensus_queue.cc:1035] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:28.456851 24444 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:28.459522 12665 catalog_manager.cc:5484] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 9 to 10, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:28.461349 14527 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:28.464262 14638 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cee7c94dd0941d99d9dd6ceb39fc190 with delete_type TABLET_DATA_TOMBSTONED (TS 8398e6f89bf5419d81ed5bed991da0cf not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41008
I20240503 08:00:28.465709 25192 tablet_replica.cc:330] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: stopping tablet replica
I20240503 08:00:28.466102 25192 raft_consensus.cc:2229] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:28.467319 25192 raft_consensus.cc:2258] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:28.467926 12946 raft_consensus.cc:2932] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:00:28.469033 25192 ts_tablet_manager.cc:1875] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:28.482935 25192 ts_tablet_manager.cc:1888] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:28.483115 25192 log.cc:1196] T 5cee7c94dd0941d99d9dd6ceb39fc190 P 8398e6f89bf5419d81ed5bed991da0cf: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5cee7c94dd0941d99d9dd6ceb39fc190
I20240503 08:00:28.483726 12658 catalog_manager.cc:4830] TS 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): tablet 5cee7c94dd0941d99d9dd6ceb39fc190 (table rebalance_test_table_6 [id=15283cbbb93542699834e6ff89a813c5]) successfully deleted
W20240503 08:00:28.489514 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.521802 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:28.524257 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.530984 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.548259 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.550863 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.560400 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.590698 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.591674 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.599079 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.607116 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.607369 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:28.609349 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.610116 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.617413 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.623090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.633909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.646739 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.650943 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.651507 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:28.656430 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.662997 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.671653 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.673084 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.673350 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.675644 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:28.690593 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.741991 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.752890 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.754110 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.768076 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.768092 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:28.769963 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.772284 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.779493 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:28.786381 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.846570 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:28.850253 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.855875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.866111 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:28.868243 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:28.875263 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:28.882565 12813 consensus_queue.cc:237] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:00:28.883901 12946 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:28.884011 14527 raft_consensus.cc:1261] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:28.884284 25272 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:28.884497 24665 consensus_queue.cc:1035] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:28.886657 24665 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.886960 14527 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.888612 12946 raft_consensus.cc:2932] T 389b40c2136d45ab86ccf49df8d1b71e P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.890039 12670 catalog_manager.cc:5484] T 389b40c2136d45ab86ccf49df8d1b71e P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 8 to 9, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:00:28.894019 14769 tablet_service.cc:1514] Processing DeleteTablet for tablet 389b40c2136d45ab86ccf49df8d1b71e with delete_type TABLET_DATA_TOMBSTONED (TS b052e89b63c24258872c832a6a0d4d4d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55922
I20240503 08:00:28.894652 25428 tablet_replica.cc:330] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: stopping tablet replica
I20240503 08:00:28.895028 25428 raft_consensus.cc:2229] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:28.895224 25428 raft_consensus.cc:2258] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:28.896181 25428 ts_tablet_manager.cc:1875] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:28.899600 25428 ts_tablet_manager.cc:1888] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:28.899693 25428 log.cc:1196] T 389b40c2136d45ab86ccf49df8d1b71e P b052e89b63c24258872c832a6a0d4d4d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/389b40c2136d45ab86ccf49df8d1b71e
I20240503 08:00:28.900331 12659 catalog_manager.cc:4830] TS b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): tablet 389b40c2136d45ab86ccf49df8d1b71e (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:00:28.901844 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:28.916240 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:28.955329 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:28.980191 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:28.982388 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:28.991557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.005841 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.009999 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.020648 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.023914 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.083436 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.107213 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.107326 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.114987 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.136502 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.137241 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.142047 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.182039 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.210336 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.214490 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.215744 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.221803 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.238173 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.238510 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.239912 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.240113 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.240128 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:29.280259 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:29.300614 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.316033 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.319696 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.323187 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.329387 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.335649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.341691 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.344928 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.351061 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.352486 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.352593 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.352854 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.369179 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.372090 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.388164 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.393483 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.403833 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.413082 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:29.443938 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:29.471413 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:29.530934 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.534852 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.544687 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.553936 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.554461 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.554612 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.578038 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.606537 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.608763 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.618806 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.619230 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.627367 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.627805 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.649271 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.679726 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.693742 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.694545 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.695078 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.696766 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.703510 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.711091 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.783139 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.788887 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.789140 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.789870 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.796106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.813980 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.819682 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.857880 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.860472 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:29.861244 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:29.865739 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.871249 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.871487 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:29.879936 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:29.883073 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.900569 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.934479 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:29.936338 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:29.966666 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:29.968293 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.970357 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.982733 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:29.987102 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.068168 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.073684 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.075810 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.086161 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.102825 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.103694 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.125134 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.134539 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.134809 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.142249 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.153102 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.155937 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.169176 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.177376 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.182893 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.196877 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.199065 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.203217 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.206779 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.211670 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.214669 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.231649 14788 consensus_queue.cc:237] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:00:30.233085 13208 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:00:30.233304 12944 raft_consensus.cc:1261] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:00:30.233810 25636 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:30.234117 25636 consensus_queue.cc:1035] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:30.255692 25976 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:30.257694 13208 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:30.262291 12673 catalog_manager.cc:5484] T 7c630e5cf7c0419cbf5d13413d22d317 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 11 to 12, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:30.262621 12944 raft_consensus.cc:2932] T 7c630e5cf7c0419cbf5d13413d22d317 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:00:30.280050 14507 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c630e5cf7c0419cbf5d13413d22d317 with delete_type TABLET_DATA_TOMBSTONED (TS 82cd33f2bae4419b946095f8eb93e1e4 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39370
I20240503 08:00:30.285151 26125 tablet_replica.cc:330] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: stopping tablet replica
I20240503 08:00:30.285567 26125 raft_consensus.cc:2229] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:30.285828 26125 raft_consensus.cc:2258] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:30.286706 26125 ts_tablet_manager.cc:1875] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:30.289717 26125 ts_tablet_manager.cc:1888] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
W20240503 08:00:30.289755 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:30.289878 26125 log.cc:1196] T 7c630e5cf7c0419cbf5d13413d22d317 P 82cd33f2bae4419b946095f8eb93e1e4: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7c630e5cf7c0419cbf5d13413d22d317
I20240503 08:00:30.290365 12659 catalog_manager.cc:4830] TS 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): tablet 7c630e5cf7c0419cbf5d13413d22d317 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
W20240503 08:00:30.303354 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.355521 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:30.375964 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.388233 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.389220 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.389294 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.390093 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.415935 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.420845 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.426115 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.432019 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.434486 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.436745 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.449378 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.451181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.459878 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.461563 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.462030 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.474368 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.490521 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.504704 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.540272 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.542758 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.550052 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.554035 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.562202 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.583127 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.584990 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.585400 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.620625 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.680058 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.687026 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:30.688657 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:30.696611 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.696854 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.696954 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.708788 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.717533 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.739295 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.743824 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.753521 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:30.757781 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:30.758708 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:30.777886 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:30.779011 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.779508 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.784584 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.826491 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:30.828600 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.831382 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:30.831467 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:30.840955 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.850057 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.851830 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:30.852561 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.874696 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.927824 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.942379 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:30.971751 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:30.975399 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:30.982147 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:30.982751 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:30.985287 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:30.986413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.006992 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:31.007383 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.011059 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.021463 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.021605 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.022143 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.022884 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:31.036947 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.048416 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.067487 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.069347 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:31.092540 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.113302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.119259 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.126643 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.130347 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.136924 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.143435 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.162349 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:31.163394 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.169226 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.175763 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.198628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.204856 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.215728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:31.216281 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:31.217171 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.227387 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:31.228091 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.242134 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:31.244617 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:31.257505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.262226 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.274911 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.277913 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.288466 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:31.298969 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.300063 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:31.308207 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 66 similar messages]
I20240503 08:00:31.370332 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.376144 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.376331 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.387913 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.395903 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.398209 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.428006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.453961 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.458772 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.473917 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.474526 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.479615 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.482228 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.483486 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.498780 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.511258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.515430 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.520308 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.520879 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:31.527357 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:31.532178 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.550495 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:31.603027 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:00:31.609205 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240503 08:00:31.645820 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:00:31.656975 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 08:00:31.658432 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.659076 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.662724 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.666990 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.672629 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:31.674949 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:31.690744 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.692447 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.693161 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.698590 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.699615 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.718454 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.719234 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.731783 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.741472 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.742322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.742995 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.753933 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:31.756744 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:00:31.763093 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.788741 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:31.794579 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.800951 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.808153 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:31.823381 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:31.850369 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:31.853590 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:31.860390 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.863919 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:31.871495 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:31.883656 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:31.888168 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.000324 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.006577 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.012969 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.013561 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.025518 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.039072 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.045933 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.048498 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.049882 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.051854 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.052096 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.057237 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.066058 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.123632 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.128070 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.134233 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.140642 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.140702 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.152659 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.161595 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.181411 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.186901 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.189949 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.192934 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:32.194216 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:32.200415 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.202584 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.228284 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.232131 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:32.298647 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:32.310022 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.310640 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.316802 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.329339 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.331945 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.347872 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.369150 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.406085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.406126 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.410315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.416913 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.429173 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.439673 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.445042 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.449199 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.449507 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.451920 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.455233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.468245 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.481339 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.495113 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:32.545302 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:32.626914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.629323 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.639439 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.643381 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.644402 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.644678 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.645560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.647106 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.650904 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.659130 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.669337 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.673427 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.681636 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.689950 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.707345 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.714322 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.723393 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.727577 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:32.738617 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:32.752360 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.787058 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.788388 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.809897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.813910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.833683 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.837205 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.843089 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.848109 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.873780 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.908674 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:32.919863 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:32.921717 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:32.930438 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:32.930461 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.940032 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:32.941370 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:32.941955 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:32.973057 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:33.030113 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.030377 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.030630 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.037955 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.047749 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.051259 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.066485 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.086289 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.104215 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.107821 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.121886 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.124341 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.128643 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.131058 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.139716 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.144008 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.155069 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.158332 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.159865 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.168355 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.207701 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:33.233773 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.259934 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:33.308681 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.314575 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.323210 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.323837 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.345599 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.346174 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.350339 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.355103 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.361351 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.367580 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.377342 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.383397 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.391855 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.394140 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.399068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.401293 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.406375 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.408855 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.417444 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.423609 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.426972 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:33.439568 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.454658 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:33.504050 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.512630 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.514537 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.515558 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:33.524142 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.526254 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:33.526717 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.527657 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.535847 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:33.557602 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.564282 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.598999 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:33.610659 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.615705 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.620142 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.624341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.639171 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.639794 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:33.648805 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:33.651103 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.651789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.653874 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.658429 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.663111 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.671634 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.686092 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.691485 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:33.710134 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.763667 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:33.775187 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.797365 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:33.802596 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.803740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.805279 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.810295 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.814014 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.847754 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.859784 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.925874 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:33.933984 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.943337 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:33.947670 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.947671 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:33.959920 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.981201 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:33.994302 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:33.995244 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:33.996038 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.005784 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.006016 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.010774 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.011570 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.073438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.073474 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.077541 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.083010 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.085915 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.090054 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:34.097199 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:34.106875 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:34.202554 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:34.210410 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.214166 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.214854 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.226974 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.227622 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.238056 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.238381 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.260839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.264981 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.272078 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.273272 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.273922 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.301860 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.303665 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:34.336483 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:34.405783 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.406433 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.416360 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.425671 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.433455 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.437060 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.445609 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.446583 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.447546 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.460461 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.476423 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.483973 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.485288 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.505409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:34.512775 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:34.591974 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.594693 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.594735 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.595065 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.606248 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.621771 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.623147 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.627264 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.640404 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.645083 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.662544 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.690088 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.694232 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.698509 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.733105 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:34.738533 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:34.740700 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.744050 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.744374 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.761276 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.774420 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.797291 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.803334 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.804190 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.809618 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.811090 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.813380 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.813688 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.819900 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:34.837563 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:34.851315 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:34.858829 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:34.875324 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.888521 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:34.890596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:34.906132 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.003515 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.003911 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.007400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.014009 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.015547 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.037741 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:35.038046 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.040511 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.089073 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.092480 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.100922 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.108600 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:35.118384 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:35.119411 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.122448 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.136333 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.138857 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.154722 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.160742 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.161471 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.167178 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.187359 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.220034 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.309280 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.314039 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.315984 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.320626 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.333549 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.335085 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.339318 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.343164 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:35.344821 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:35.354156 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.364620 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.373111 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.376019 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.398557 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.404233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:35.413422 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.441771 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.454460 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.464586 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.464697 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.467339 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.471904 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.478967 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.527662 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.542609 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.542860 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.544549 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.546607 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:35.557864 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:35.564915 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.575131 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.589510 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.591399 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.602018 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.607601 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.618036 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.622659 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.636507 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:35.665100 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.693362 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.694080 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.696741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.697057 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.697398 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.710598 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.736371 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:35.738581 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.832374 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.850379 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:35.867820 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.869608 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.875327 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.877538 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.879760 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:35.885026 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.887467 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:35.895888 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.899067 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:35.901108 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:35.919998 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:35.922513 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:35.922736 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.923125 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.936286 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:35.967566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:35.971326 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:35.976789 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.986335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.001410 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.006546 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.007263 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.022912 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.033501 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.049724 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.051314 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.051400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.051400 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.051477 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.053898 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.057420 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.061187 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.064620 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.071436 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.071749 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.082502 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:36.092669 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:36.094640 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.100515 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:36.116329 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:36.189229 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.294191 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:36.297969 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.301764 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.307716 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.308321 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.309880 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.335853 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.336320 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:36.339025 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:36.353134 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 08:00:36.414625 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:36.430395 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.436023 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.437631 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.439493 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.441546 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.472124 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.485576 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:36.495064 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.507692 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:36.514357 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.523830 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.525002 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.528188 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.528486 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.537000 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.560375 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.567975 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.573038 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.576296 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.584174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.588886 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.608297 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.619248 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.622819 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.632539 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.641742 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:36.642076 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:00:36.642338 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:36.647682 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.651448 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.657168 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.657891 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:36.743177 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:00:36.743916 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:36.786628 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:36.788831 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:36.793228 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.799644 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.800447 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.815114 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.815866 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.820927 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.821583 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.827009 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.829130 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:36.845260 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.849946 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.854910 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.871136 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:36.892518 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:00:36.933840 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:36.948385 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:36.953994 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:36.957264 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:36.958948 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:36.969874 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:36.972524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:36.992550 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.004720 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:37.007838 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:37.012176 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.012959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.013388 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.018833 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.043561 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.045346 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.104601 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.111105 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.115247 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.116039 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.130863 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.135253 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.158380 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.207958 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.217332 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.220343 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.238224 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:37.248343 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 08:00:37.250531 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.265429 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.292631 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.293057 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.295671 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.303395 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.305586 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.306046 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.316002 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.319442 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.332726 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.335163 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.347916 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:37.364228 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:37.369467 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.370455 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.384528 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.393488 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.476284 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.476534 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.482077 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.486372 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.489704 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.502575 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.517925 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.521770 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.522980 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:37.526665 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:37.536465 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.553498 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.555430 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.557929 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:37.573170 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:37.584048 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.649612 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.656682 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.658389 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.658746 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.676926 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.679837 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.687245 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.734944 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.745914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.755602 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.755764 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.770308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.785022 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.788401 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:37.803015 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:37.841701 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.842072 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.852782 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.865505 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.871366 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.881255 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:37.912698 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:37.924765 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:37.925216 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:37.940452 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:37.958199 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:37.968356 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:37.968436 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:37.987087 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.000698 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.018543 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.036938 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.041175 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.050771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.066548 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.067528 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:38.090544 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:38.097447 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:38.111769 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.157068 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.158514 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.162554 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:38.171679 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.184957 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.185675 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.187090 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.211392 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.264178 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.271950 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.283869 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.294451 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.304296 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.313668 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.315531 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.328501 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.333190 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.342506 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.348664 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.348979 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.349411 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.357290 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.358011 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.361788 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.393764 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.394677 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.401165 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.405290 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.426883 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:38.449524 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.449918 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.461351 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.481966 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:38.529388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.530506 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.535409 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.538005 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:38.547910 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.554692 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:38.557706 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:38.561187 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.562129 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.588603 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.605196 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.627511 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.627990 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.633914 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.634801 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.649353 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.653999 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.654670 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.674503 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.676126 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.685346 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.687669 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.687902 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:38.688153 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.694654 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.709254 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.709473 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.715807 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.742520 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.745625 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.750821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:38.828878 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.838819 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.842005 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.844139 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:38.844393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:38.850989 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:38.863860 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.870026 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.870689 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:38.984041 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:38.987337 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:38.990105 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.990276 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:38.991631 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:38.999001 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:39.006055 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:39.009104 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.015952 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.017637 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.018939 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.029217 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.035354 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.038123 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.051585 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.062657 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:39.083704 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:39.153558 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.159492 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.159829 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.161943 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.163934 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.189628 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.191527 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:39.218290 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:39.266433 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.290069 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.290125 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.291146 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.295636 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.298970 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.302999 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.303099 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.307356 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.329674 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.333750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.354069 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.369613 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.375762 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.392772 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.393414 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.396638 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.397043 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.405378 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.419405 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.427953 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:39.430622 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:39.492338 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:39.520834 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.522488 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.536252 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.539059 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.542275 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.559696 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.562261 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.573271 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.585827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.589643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.593936 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.596830 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:39.600124 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:39.607864 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.621685 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.662009 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.669960 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.672454 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.675285 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.684347 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.689781 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.702694 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.738593 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.740327 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.743986 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.760833 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.765285 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.771334 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.781949 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.843937 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.846017 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:39.854636 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:39.863067 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.865432 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.876590 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.883800 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:39.922549 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:39.953387 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:39.957806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:39.961652 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:39.963172 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:39.965749 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.000509 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.003391 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.007954 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.013384 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.018774 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.019364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.023766 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.034097 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.048027 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.108681 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.118220 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:40.120363 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:40.126297 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.138265 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.140478 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.158743 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.163089 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.181653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.193722 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.214752 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:40.217269 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:40.219174 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.221868 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.223310 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.237880 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.268095 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.271121 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.272238 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.282068 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.283020 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.288551 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.291474 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:40.338641 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:40.356248 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.359511 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.360316 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.366111 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.366369 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.374274 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.387151 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:40.458858 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:40.467871 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.485462 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.487143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.487515 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.493539 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.494769 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.499167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.518739 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.529330 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.536206 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:40.537892 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:40.540709 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.559075 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.574877 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.578461 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.593017 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.602399 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.609526 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.609640 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.611856 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.619689 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.639006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:40.675369 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:40.754526 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.761134 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.762413 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.768087 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.773887 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.776145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.783211 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.783468 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.799660 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.799970 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.809556 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.811203 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.816114 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:40.840543 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:40.846801 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:40.856710 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:40.862715 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:40.862888 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:40.863386 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:40.875890 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.877305 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:40.884137 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:40.928278 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.982517 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.992628 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:41.037309 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.039851 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:41.041325 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:41.043174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:41.046537 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.051118 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.058385 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:41.063470 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.073844 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.085602 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.087481 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.112524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.119083 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.125485 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.129452 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.139216 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.142701 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.143383 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.150125 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.153548 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.157009 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.170686 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.207136 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.207551 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.207787 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.207898 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.209484 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.214380 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.228303 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.229730 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.233510 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:41.309000 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:41.339643 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.360962 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.361845 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.362694 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.377260 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.386122 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.389091 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.391017 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.391641 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.403226 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.411126 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.412863 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.420236 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.436123 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.447608 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.497054 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.497340 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.499166 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.503942 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:41.505424 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:00:41.508819 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:41.511070 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.517563 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.525097 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:41.539810 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:41.609385 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.642896 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.647529 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.648862 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.652427 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:41.653414 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:00:41.663033 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:41.667430 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.673669 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.680110 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.691416 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.700951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.701191 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.707875 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.708470 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.709429 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.722318 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.722892 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.732450 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.736682 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.739950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.741034 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:41.745813 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.795521 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:41.796211 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.879927 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.881635 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.885438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.886396 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:41.889703 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:41.894418 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:00:41.899015 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 08:00:41.907150 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:41.909941 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:41.980983 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:41.995776 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:41.998159 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.005100 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:42.008253 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:42.008509 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.011080 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.012300 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.013821 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.015679 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.018458 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.034969 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.051841 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:42.059970 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:00:42.068199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.075223 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.078624 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.093631 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.098227 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.110033 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.116794 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:42.125908 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:00:42.129257 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:42.132876 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:42.134713 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:00:42.135190 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.226466 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.228961 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.231194 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.231537 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.236311 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.271729 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.303479 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.324340 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.345417 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.350158 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.351935 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.358556 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.371938 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.374670 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.375938 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.390338 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.396122 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.401168 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.415095 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.426522 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.447430 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.454741 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.461084 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.461508 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.468322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.470029 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:42.480151 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:42.503018 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:00:42.504844 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.512097 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.532003 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.535228 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:42.540433 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:42.541538 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.543534 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.544111 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:42.563915 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:42.567174 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.573426 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.657239 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.657657 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.668537 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.673503 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.683996 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.687413 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.706893 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.713320 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:42.717804 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:42.719455 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.724839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.725878 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.726541 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.749832 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.750838 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.822511 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.850031 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:42.850530 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.873843 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.876286 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.879904 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.880728 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.901454 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.903101 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:42.906908 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.915858 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.919872 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.925961 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:42.939090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.022853 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.033859 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.034838 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.050601 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.065919 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.068847 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.071511 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:43.074229 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.078195 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:43.097481 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.099247 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.101375 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.113616 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:43.114571 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:43.121421 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.122197 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.142200 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.171130 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.186137 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.195652 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.210001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.215898 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.222841 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.223531 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.235777 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.242085 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.246599 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.247669 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.252195 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.271411 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.288051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.298074 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.344139 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.381899 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.400995 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.403980 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.411480 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.415764 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.421128 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.430794 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.433882 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.442196 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.444142 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.444518 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.448834 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.450264 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.450740 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:43.455124 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.481247 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:43.495095 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:43.534891 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.539238 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.539597 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.553339 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.558357 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:43.563447 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:43.564765 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.576081 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.597064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.604394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.605441 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.606945 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.610669 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.634761 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.641628 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.685361 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.721531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.723047 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.723502 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.743983 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.744271 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.745425 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.754205 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.767952 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.771200 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.777024 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.783618 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.784926 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.787576 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.792996 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.793835 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.821014 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.899731 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.900974 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:43.910243 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:43.928614 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:43.935515 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.938069 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.955906 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:43.961833 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:43.964303 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:43.966472 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.967979 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.986656 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:43.988493 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:43.998119 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.017058 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:44.068352 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:44.071221 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:44.071377 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.080435 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.080658 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.081350 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.097358 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.108893 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.116515 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.120908 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.127903 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.136518 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.139964 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:44.140223 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:44.144748 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.150909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.163081 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.239071 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.243119 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.251837 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.254070 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.256666 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.272159 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.278318 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:44.280663 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:44.312803 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.324483 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.325143 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.326400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.326696 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.341423 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.371282 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.446695 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.453572 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.454677 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.456568 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.471024 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.483814 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.515349 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.518632 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.521616 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.528227 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.528617 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.532461 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.540032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.547609 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.547900 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.555583 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:44.557287 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:44.563632 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.570780 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.575079 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.578493 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.611297 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:44.629324 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:44.636772 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.640394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.641780 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.647158 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.647553 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.648068 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.656682 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:44.660451 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:44.736148 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.741178 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.753742 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.757635 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.768816 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.771003 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.776969 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.812326 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.815843 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.818814 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.829097 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.830718 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.840718 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.861780 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.919457 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:44.928639 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:44.930095 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:44.933977 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:44.936380 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:44.959040 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:44.987434 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.010604 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.016491 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.022135 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.027869 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.029661 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.030316 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.035984 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:45.042634 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:45.077205 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:45.113341 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.120353 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.129709 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.134723 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.134721 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.139223 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.144639 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:45.151502 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.155360 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.162418 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.180871 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.214623 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.218609 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.219434 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.220326 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.228797 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.239487 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.245667 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.261935 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.263262 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.263509 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.278264 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.393095 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.397487 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.406090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:45.406497 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.409799 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.429239 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.443439 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.452373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.452733 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.454486 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.460043 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.463171 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.465234 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.478291 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:45.491369 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.506592 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.508347 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.513413 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.524996 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.533164 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.537285 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.538563 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.565771 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:45.576437 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:45.619678 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.624001 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.625032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.625910 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.630110 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.630434 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.658041 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:45.677706 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:45.694501 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.704110 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.708925 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.718516 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.718585 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.719960 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.741245 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.769822 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.771230 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.789278 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.793226 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.795145 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.796837 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.817075 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.822216 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.883651 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.890159 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:45.905644 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.907577 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:45.909049 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.912348 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:45.916735 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:45.928218 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.943832 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:45.946385 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:45.950829 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:45.952769 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.980692 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:45.984875 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:45.986083 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.002076 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:46.003269 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:46.051559 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.053398 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.060345 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.060810 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.068977 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.069008 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.086942 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.161377 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.162838 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.178795 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.180871 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.184053 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.199623 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.199975 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.203087 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.203183 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:46.220571 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.224671 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:46.238637 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:46.247187 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.251878 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.255362 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.265815 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:46.274864 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.276414 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.343744 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.350382 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.351147 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.354717 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.358685 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.367138 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.367630 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.379236 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.389917 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.391885 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.398515 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:46.400225 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.405707 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.419376 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.428629 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:46.442466 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:46.495191 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:46.504783 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.506186 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.508498 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.519838 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.537130 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:46.545538 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:00:46.546799 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.548185 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.556439 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.563494 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.564512 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.566964 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.574108 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.575579 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.580629 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.583098 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:46.666759 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:46.680934 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.687175 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.690233 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.701349 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.703171 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.711980 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.734357 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.736779 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:46.740170 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:46.741251 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.747363 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.747529 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.749583 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.760931 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.768597 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:46.807161 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:46.849520 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.850164 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.850512 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.851119 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.851476 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.856535 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.876109 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:46.933009 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.933272 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:46.935520 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:46.936954 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:46.941310 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:46.959681 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:46.991309 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:47.001824 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:00:47.002414 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:47.022030 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:00:47.039666 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.041738 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.042685 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.045068 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.050231 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.052644 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.066282 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.069655 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.069916 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.083062 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.094668 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.095860 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.110769 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.117998 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:47.120115 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:47.159315 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:00:47.200059 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.201752 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.206524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.208822 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.217744 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.224104 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.232538 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.294003 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.297551 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.298133 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.299515 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.304685 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.321457 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.341573 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.372217 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.373409 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.378365 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.382974 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.383644 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.384603 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.396515 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:47.413797 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:00:47.439244 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:00:47.450596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.454190 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.457348 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.460848 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.464531 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.467231 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.499045 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:47.525166 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:47.568599 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:47.580257 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.580860 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.588814 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.604148 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.606570 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.613946 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.618559 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.626338 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.627957 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.651464 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:47.659603 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:47.664844 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.668813 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.672099 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.680066 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.716513 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.722594 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.724093 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.747402 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:47.751979 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:00:47.752884 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:47.753562 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.754539 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.763471 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.822801 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.825047 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.829836 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.831348 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.848453 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.848544 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.875526 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.895597 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.901865 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.917413 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.921382 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:47.924194 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.933617 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:47.951931 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:47.952138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:47.955049 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:47.956737 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:47.962677 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.965612 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.983352 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:47.988663 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:47.989427 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:48.040771 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.110330 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.114296 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.119172 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.126315 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.127219 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.132258 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.164216 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:48.193011 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:48.204408 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.224483 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.224663 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.228777 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.234534 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.237016 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.240680 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:48.249557 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.258697 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.259145 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.265519 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.266319 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.273926 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.281256 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.285537 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.295920 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.307760 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.311327 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.318898 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.320160 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.359936 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.364001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:48.366521 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.420718 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.422243 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.424279 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.424590 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.443866 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.454656 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.459074 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.507294 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.508266 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.508772 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.523784 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.526118 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.532474 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.551024 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:48.553314 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:48.644927 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.645182 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.652259 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.652503 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.661192 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.662328 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.666538 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:48.691845 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.720845 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.748422 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:48.764288 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.787699 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.796106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.797490 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.821004 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.834749 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:48.836469 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.843744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.844198 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.853123 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.856412 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.862798 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.868165 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.871059 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.877836 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.882382 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:48.887989 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.892807 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.895077 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:48.897276 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.902509 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:48.909526 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.910029 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:48.912508 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:48.932224 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:48.945492 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:48.956168 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:48.966477 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:48.978796 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.981606 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:48.989121 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.001011 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:49.049360 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:49.066164 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.066936 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.080029 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.080956 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.083873 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.106837 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.109719 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:49.147547 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:49.167336 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.170202 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.173679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.177706 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.180094 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:49.191816 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:49.195530 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:49.202530 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:49.207684 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.238219 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.240250 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.240835 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.245816 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.246307 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.246380 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.247123 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:49.323954 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 08:00:49.400090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.400943 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.410318 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.411583 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.414885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.415258 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.428576 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.429957 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.434358 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.450790 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.459411 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:49.463835 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:49.473428 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.475091 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.478896 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.485180 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.491240 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.499150 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.500325 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.525738 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.541121 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.551730 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.577688 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.595904 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.600872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.610059 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.610131 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.610234 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.635107 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:49.689662 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:49.700605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.705580 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.713956 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.716152 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.717584 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.745257 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.750114 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:49.758028 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.761065 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.767136 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.768432 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.775125 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.782598 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.786670 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:49.907289 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:49.907821 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.911933 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:49.917140 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:49.919142 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:49.923763 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:49.924222 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:49.927574 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:49.932436 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:50.036444 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.039911 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.040910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.041544 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.045431 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.062176 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.068866 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.082507 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.083945 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.084200 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.103437 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.104943 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.111183 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.125702 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:50.126308 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:50.127888 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.163559 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.165714 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.174261 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.175745 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.198501 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.199136 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.199549 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.203408 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.204071 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.215548 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.221814 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.233420 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.249439 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:50.298599 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:50.329121 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.330617 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.345531 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.345804 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.346639 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.355886 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.355955 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.406364 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.407370 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.407776 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.408751 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.418239 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.418437 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.418874 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:50.462853 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:50.559918 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.561445 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.569360 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.574746 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.580250 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.582661 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:50.591617 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:50.594514 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.598017 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.600982 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.605453 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.621004 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:50.641425 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:50.644912 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:50.650663 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:50.660589 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.667313 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.707332 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.723867 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.749012 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.759255 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.759563 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.774449 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.776376 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.777313 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.789853 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.790446 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.791299 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.798233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.799973 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.800379 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.810597 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.814388 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.825193 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:50.828248 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:50.850358 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.858354 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.864017 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.868535 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:50.880817 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:50.949890 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:50.954644 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.964042 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:50.964697 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.965973 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.976986 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:50.995231 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.054114 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.060438 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.061868 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.062500 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.065033 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.071179 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.083245 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.123059 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.123634 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.136216 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.138751 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.141530 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:51.165798 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.181058 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.183094 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:51.194973 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.195066 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.213032 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:51.236768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.240082 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.250545 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.268944 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.269055 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.278612 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.293241 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:51.303694 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.342782 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.362637 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.379901 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.386672 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.387995 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.399418 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.400538 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.407069 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.410382 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.410640 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.420974 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.435128 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.436591 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:51.439899 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.453338 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.456807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.465366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.465612 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.465623 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.473814 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.477304 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.502378 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:51.502527 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:51.565299 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.567309 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.570322 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.571847 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.573160 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.579074 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:51.582453 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 08:00:51.583081 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.596818 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.603680 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:51.610816 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.657047 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.658354 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.662313 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.667104 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.671964 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.681864 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.694267 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:51.695727 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:51.774621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:51.787997 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 08:00:51.798466 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.799142 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.809088 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.817651 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:51.820927 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.828868 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.855157 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.860697 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:51.864249 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:00:51.868022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.875465 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.877856 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.878077 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.893901 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.928138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:51.929618 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:51.930744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.933214 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:51.956153 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:51.957257 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:51.957401 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.012979 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:52.017511 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:00:52.018159 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:52.019793 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.024518 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.029480 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.041097 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.042263 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.042479 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.056253 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.061540 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.081928 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.083791 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.089649 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.107259 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:52.109627 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:52.112905 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.183480 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:52.183840 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:52.188726 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.190323 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.195730 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.195803 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:52.197299 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:52.212018 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.226511 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.283386 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.283448 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.292804 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.292953 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.317245 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.325821 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.333935 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.422224 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.424525 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.425650 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:52.427486 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:52.435083 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.444759 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.473796 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.476329 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.487498 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.490505 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.494469 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.502591 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.520475 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.521823 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.522264 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.522637 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.525094 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.532959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.533277 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.542888 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.571002 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:52.573086 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:00:52.575381 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:52.623100 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:00:52.632795 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.641393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.641505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.642095 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.646996 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:52.655243 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:52.661119 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.661381 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.671723 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.673949 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.686596 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.705265 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.709492 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.710350 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.712451 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.813925 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.818506 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:52.820394 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:52.832999 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.835268 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.840416 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.842413 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.854460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.879724 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.884420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.886579 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.900380 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.907917 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:52.909230 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:52.922008 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.931094 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:52.943423 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.945008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:52.955197 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:52.955946 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:52.963323 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:52.984663 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:00:52.985785 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:52.989758 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:53.019948 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.069025 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.086305 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.101521 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.106514 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.107471 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.111363 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.115643 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.119385 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.123524 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.128549 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.133285 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.148694 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.149926 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.172122 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.172748 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.176334 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.179286 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.195089 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.198315 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.206686 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.209872 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.211920 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:53.224372 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:53.254217 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.338986 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.340817 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.345803 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.348575 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.362850 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:53.373814 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:53.374583 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.388545 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.429323 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.437840 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.440433 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.447077 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.447374 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.449242 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.453333 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.458389 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.462230 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.496526 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.499117 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.503822 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.504101 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.515838 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:53.524215 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.585184 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.587225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.597926 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.609079 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.613368 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:53.618728 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:53.627188 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.642506 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.695209 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.697962 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.699226 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.703187 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.718417 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.731755 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.742427 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:53.774080 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.789029 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.805445 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.809010 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.811172 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.816567 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:53.823611 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:53.835870 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.836921 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:53.868945 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.870400 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:53.878857 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.889559 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:53.893131 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:53.906761 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:53.907554 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:53.911499 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:53.949386 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:53.960317 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.965021 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:53.999425 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.000731 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.006553 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.011886 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.024010 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.025094 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.041554 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.059365 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.077076 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.080999 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.083962 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.093451 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.100852 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:54.124687 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:54.124773 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:54.152671 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:54.157477 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.172744 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.173621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.174086 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.191628 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.193462 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.217665 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:54.224361 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:54.317184 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.320861 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.339274 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.343318 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.345149 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.350239 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.360726 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.370244 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.372237 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.376992 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.396440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.398620 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.400741 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.403237 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.403277 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.409193 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.417397 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.419818 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:54.420058 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:54.426898 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 08:00:54.442152 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.452181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.453336 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.528471 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.534312 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.539165 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.545930 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.556823 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.572587 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.579717 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.580421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.581148 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.583338 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.593446 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.594190 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.620155 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.624608 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:54.688990 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:54.720162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:54.720660 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:54.724881 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.726399 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.752606 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.753243 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.753793 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.754034 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.762259 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.774360 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.777451 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.785805 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.792243 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:54.795536 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:54.802814 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.826196 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.930148 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.932910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:54.948299 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:54.976981 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:54.979192 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.007311 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.014981 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.019912 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.022655 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.031893 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.038106 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.038767 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.063299 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.069942 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.089572 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.099610 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.100741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.101320 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.110071 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.110733 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.122577 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.133054 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:55.146620 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:55.157488 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.160985 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.163581 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.163708 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.168401 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.169445 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.171150 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.187078 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.215641 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.226390 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.228307 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.228741 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.238325 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:55.269883 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.393886 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:55.397217 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.404878 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.412737 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.413679 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.417043 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.417649 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.419198 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.425418 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.445842 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.459786 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.466573 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.479938 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.480692 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.484040 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.485384 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.495342 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.496513 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.496904 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.522681 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.533130 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.546633 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.631701 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.650178 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.650177 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.664881 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:55.675964 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:55.682018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.683586 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:55.685174 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:55.692829 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.700157 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.701244 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.717753 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:55.722280 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.722741 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.729975 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.733973 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.738513 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.741025 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.746639 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.754864 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.788436 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.811877 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.812705 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:55.823254 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.828358 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:55.875232 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.875594 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.877221 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.880970 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.884565 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.889238 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:55.891750 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:55.901881 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.960675 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:55.960832 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:55.962313 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:55.964532 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:55.975543 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:55.999807 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.015400 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.063875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.069314 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.069489 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:56.071110 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.071449 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.075286 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.080072 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.088359 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:56.113732 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:56.167397 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.201614 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.203435 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.211685 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.217868 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.226704 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.230501 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.234622 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.242897 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.267125 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.274153 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.277606 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.294250 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.295869 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.311197 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.315678 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.315749 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.318516 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.323711 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.347651 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.348527 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.363035 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:56.365326 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.365967 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:56.368563 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:56.371331 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.372035 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.388469 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.400636 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:56.429939 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.435357 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.443739 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:56.481323 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.503763 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.505728 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.508828 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.511857 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.520166 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.521750 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.522826 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:56.563235 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:56.569504 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.574617 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.582401 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.593453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.601210 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.601936 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.613598 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.674608 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.680245 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.681088 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:56.688457 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:00:56.690100 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:56.690541 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.696187 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.701340 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.715039 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:56.743356 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.745723 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.747994 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:56.798736 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 08:00:56.803707 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.815521 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.819332 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.828594 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.842201 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:56.843235 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.847426 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.847483 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:56.852195 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.853665 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:56.861815 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.867285 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.893203 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.912700 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.957942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:56.960862 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:56.961647 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:56.987375 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.000114 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.007948 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.012696 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.019596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:57.020284 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.021456 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 08:00:57.021546 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.021996 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:57.022622 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:57.032378 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.038421 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.067145 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.070667 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.102612 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.103698 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.114471 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.115008 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.118078 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.121881 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.146277 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.195302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:57.202152 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:57.211668 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.223515 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.225994 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.226032 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.237434 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.248828 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.283767 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.291067 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.304242 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.308123 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.313491 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.313715 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.324144 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:57.356634 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:00:57.383814 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.384598 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.389621 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.397821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.400144 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.401937 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.422652 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:57.435308 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:57.442513 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:00:57.458092 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.461344 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.474581 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.501636 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.502750 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.509725 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.510668 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.526800 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.534605 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.541186 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.541965 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.549578 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.570669 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.587394 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.631503 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.635474 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.636770 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.638434 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.645475 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.652333 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.664929 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:57.683475 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:57.741746 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.746376 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.748533 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.757290 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.760759 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.763664 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:57.785104 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:57.786684 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.791899 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.805080 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.819936 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.823187 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.824154 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.827791 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.829042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.861037 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:57.864854 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:57.874207 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:57.875686 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:57.879580 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:57.890159 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:57.890254 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:57.912845 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:00:57.948354 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:00:57.961220 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:58.008917 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.017369 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.018086 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.018754 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.021587 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.030697 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.034519 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.044891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.048286 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.069474 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.078048 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.078516 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.113457 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.116147 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.119401 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.133807 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.133822 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.137677 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:58.137943 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:58.141232 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.156275 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.157101 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.260578 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.264022 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.265686 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.270323 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.276373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.277766 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.281832 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.286528 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:58.291361 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.293030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:58.295894 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:00:58.296608 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.311830 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.327423 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.327764 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.329018 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.335278 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:58.423445 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.424490 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.473944 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.489804 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:58.502010 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:58.513234 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:58.516566 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.518179 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.535368 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.538142 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.539333 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.546447 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.548915 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.550813 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.559074 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.561213 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.576885 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.577293 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:58.582473 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.583374 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.599731 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.635274 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.635974 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.636844 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.639148 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.639317 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.641894 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.643818 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.654120 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.659308 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.668318 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.669874 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.689672 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:58.712523 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.770229 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.771250 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.776651 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.782058 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.785393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:58.793488 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.806888 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.857712 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.859249 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.868916 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:58.871433 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:58.871513 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.871758 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.872133 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:58.895813 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:58.908486 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:58.930651 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:58.941519 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:00:58.942056 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:58.955211 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.965278 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:58.983021 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:58.986912 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.006208 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:59.011322 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:59.070456 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:59.074954 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.088754 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.097398 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.106770 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.111748 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.111760 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.112136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.117921 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:59.119532 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:59.130038 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.137076 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.171202 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.172192 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.173074 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.174506 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.179687 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.181840 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.184032 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.200093 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.229802 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.231132 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.235870 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.301126 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.303509 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.304477 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.305832 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.310770 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.314988 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:59.322147 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:59.340442 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.391892 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.392239 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.393857 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.394551 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.408962 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:00:59.413225 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 08:00:59.431066 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.434706 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.447057 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.461172 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.469494 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.472702 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.483508 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.486460 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.492343 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:59.503948 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:00:59.558377 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:00:59.595218 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.600098 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.611292 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.640651 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.640823 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.644767 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.655611 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.668642 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.669999 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.675379 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.676982 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.682456 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.690369 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.692683 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.717358 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.742807 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.747059 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.748270 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.759902 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:00:59.766595 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:00:59.769053 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.773910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.793359 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.793679 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.803192 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:59.807309 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.817678 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.822404 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.826956 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.909507 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:59.911507 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:59.919070 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:59.921519 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.933210 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:59.949537 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:59.985407 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:00:59.996042 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:00.002832 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:00.009927 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.009908 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.021162 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.023414 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.025080 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.050037 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.056051 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.060983 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.061159 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.073137 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.076874 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.090453 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.090703 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:00.092532 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:00.095186 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.172286 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.172986 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.176805 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.193702 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.196154 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.207098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.208701 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.210978 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.212594 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.213428 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.222481 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.242941 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.268110 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.272336 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.355794 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.364287 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.365309 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.366005 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.387368 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.387946 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.399931 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:00.442468 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:00.460814 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.462744 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.467115 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.470934 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.484424 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.494446 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.504148 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.507514 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.509130 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.516029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.518810 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.542989 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.543313 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.562142 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.659497 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.667081 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:00.679642 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:00.689103 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.691671 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.716133 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.727373 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.729743 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.740047 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.741778 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.742659 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.757223 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.764122 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.776486 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.785022 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.810976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:00.818635 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:00.820923 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:00.831507 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:00.838482 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.838738 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.850174 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:00.850839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.851928 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.854653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.860919 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.881196 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.883412 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:00.889575 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:00.892551 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.904827 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:00.918319 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:00.931854 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:00.994693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:00.995433 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:00.995708 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:00.995790 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.005005 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.007628 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.046707 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.075354 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.077353 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.079788 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.084151 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.098471 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.098714 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.099228 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.152150 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.154770 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.162003 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.170964 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.176049 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.183779 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.188541 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:01.199448 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.199798 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:01.212249 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.224417 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.235057 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:01.259182 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.265489 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.274317 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.283641 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.295783 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.295922 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.299865 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:01.342372 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:01.350036 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.350705 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.355415 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.366420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.367278 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.372342 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.394793 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.457247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.459654 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.469899 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:01.480690 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:01.485504 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.485785 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.487630 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.489305 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.504009 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.504279 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.505910 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.522012 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.539044 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.544421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.546026 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.550726 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.555379 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.557608 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.583371 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.588140 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:01.608912 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:01.624966 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.639663 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.693387 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.697963 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.705320 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.706017 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.713081 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.725540 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.747273 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:01.760499 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:01.774658 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:01.783197 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.789831 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.790118 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.795949 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.799095 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.800904 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:01.808135 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:01:01.813380 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.864879 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.867702 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.870298 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:01.881646 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:01.893774 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:01.893818 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.908720 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.909425 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:01.918388 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:01.938505 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:01.938839 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:01.954500 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:01.956167 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.966061 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:01.967864 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:01.981994 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.027148 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.037716 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.039248 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.039662 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.042004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:02.047706 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:01:02.048431 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:02.056078 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.069326 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.076258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.107069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.107795 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.123104 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.125303 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.135689 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.147361 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.230819 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.233485 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.233493 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.234774 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.240371 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.245029 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.254428 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.255097 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.267550 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.274405 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.281760 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:02.294454 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:02.318137 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.323608 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.327864 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:02.365275 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:02.388476 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.390224 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:02.393590 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:02.404697 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.404821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.407577 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.426249 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.428454 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:02.451965 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240503 08:01:02.464474 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.471832 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.475869 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.484012 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.484689 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:02.486447 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:01:02.489612 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.507009 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.556540 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.557924 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.564553 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.566331 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.566861 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.579983 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.586030 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.646572 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.648362 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.648881 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.656342 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.658200 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.661137 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.678395 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:02.713271 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.749424 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:02.762810 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.768003 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.775730 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.781117 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.787730 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.801237 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:02.806059 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.809362 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.811386 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.819865 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.827451 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.828292 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.841950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.845281 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:02.854611 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:02.894796 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:02.897253 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.903061 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:02.915782 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:02.921005 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:02.924109 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:02.927322 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:02.995697 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:03.010140 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.011653 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.012897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.030753 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.031100 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.047142 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.048012 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.109345 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.117214 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.125000 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.128208 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.132440 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.142177 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.145341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.173645 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:03.176914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.186993 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.204185 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.205209 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.205632 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.214782 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.227234 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:03.245262 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:03.312570 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.316287 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.319864 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.320335 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.342459 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.347645 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.352957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.360329 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:03.374070 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.376638 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.383654 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.408056 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:03.414486 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:03.418654 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.429334 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.430017 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.450356 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.460306 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.461299 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.465094 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.472724 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.485911 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:03.488713 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.494083 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.503772 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.579613 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.585067 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.590214 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.591455 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.592171 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.611631 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.632797 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.646528 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.650315 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:03.651998 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.670286 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.675913 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.679072 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:03.685223 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.685935 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.696018 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.696689 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.713398 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.725857 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.735038 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.735448 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.752310 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.755941 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.775375 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.779079 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:03.780133 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.786439 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.803678 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.804445 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.804879 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.805989 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.813846 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.816860 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.835443 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:03.848379 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:03.852859 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:03.895730 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.897507 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:03.904065 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:03.910218 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:03.920970 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:03.922600 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.951045 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:03.988035 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:03.990897 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.009768 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.017259 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.020963 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.029385 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.032080 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.059381 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.059940 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.060139 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.083216 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.083981 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.088636 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.091848 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:04.110419 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:04.219835 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.220364 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.221012 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.222045 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:04.251483 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:04.265161 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.266765 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.268378 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.270397 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.270426 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.271447 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.305966 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.315949 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.316452 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.316713 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.317114 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.328837 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.329151 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.339223 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.345841 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.354955 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:04.364836 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:04.365356 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.432554 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.436311 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.437387 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.443080 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.458308 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.461452 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.473337 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:04.501099 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:04.546778 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.552692 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.558193 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:04.568317 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:04.579097 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.581789 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.587972 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:04.592648 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:01:04.609367 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.615494 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.625414 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.628518 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.634804 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.636788 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.651358 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.655107 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.659067 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.662585 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.668144 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.681596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.718751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.723599 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.727420 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.757406 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.757406 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.759423 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.775390 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.785797 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:04.790366 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:04.797250 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.801615 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.863620 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:04.865320 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:04.865744 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.867511 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.875540 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.875993 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.892254 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.899442 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:04.929431 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:04.949877 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:04.952261 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:04.960251 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:04.969540 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:04.973373 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:04.975214 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:04.984520 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.063405 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.067771 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.070466 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.073004 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.075088 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.076714 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.105044 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.149353 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.149993 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.154765 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.168833 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.180023 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.195315 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.203449 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.210703 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.212049 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.214174 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.222031 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.223166 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.224429 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.242003 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:05.320019 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:05.360985 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.367537 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.371340 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.376897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.380968 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.397569 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:05.401616 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:05.429525 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.442386 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.443058 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.448944 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.456846 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.465351 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.470579 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.472891 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.475780 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.479588 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.480238 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.491402 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.497072 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.512387 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:05.525243 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:05.544761 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:05.563319 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:05.586910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.590221 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.598495 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.609846 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.615585 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.617301 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.624773 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.664306 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.669584 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.676116 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.681252 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.690001 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.703857 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.714591 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.716076 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.721868 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.739116 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.741520 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.754215 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.754587 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.760156 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:05.764571 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:05.872128 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:05.884300 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.891129 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.893599 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:05.898591 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.900298 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.908139 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.911128 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.919780 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:05.920128 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:05.928195 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:05.953157 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:05.957512 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.957758 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:05.958336 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:05.987123 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:05.996271 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.007431 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.022030 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.026754 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.035467 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.043329 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.053167 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.079502 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.081485 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:06.087865 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.093437 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.093691 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.109282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.109819 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.109905 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:06.157156 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.170331 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:06.176831 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.191051 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.213244 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.220134 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.223122 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.223927 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.243914 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.246160 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.259230 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.280666 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.288415 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.288940 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.299976 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:06.310071 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.310631 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.315212 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.320875 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.321792 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:06.333788 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.344115 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.363168 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.364102 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.378481 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.393407 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.395700 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:06.408946 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:06.475047 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.475740 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.478718 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.487098 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.488143 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.498548 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.541908 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:06.582752 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:06.585393 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.594396 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.594481 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.606817 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.620571 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.625636 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.631914 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.632237 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.635273 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.637802 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.647142 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.651880 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.658258 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.664620 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.664820 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.681665 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.690106 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.708971 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.710013 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.723605 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.729058 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:06.736104 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.836345 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.837699 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:06.840989 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:06.845336 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.847182 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.849365 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.850957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:06.863387 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:01:06.866153 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:06.911250 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.912693 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:06.933024 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.933092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:06.936208 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:06.940025 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.942072 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:06.952463 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:06.966372 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.973763 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.984407 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:06.986996 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:06.993664 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:06.999962 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.016223 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.022192 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.022614 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.029503 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.036264 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.042757 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.063355 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.072188 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.086045 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.158759 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.167949 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.170607 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.174638 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.175022 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:07.181000 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:01:07.181702 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:07.199041 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:07.227995 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 08:01:07.228433 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.259032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.280841 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.281666 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.282106 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.282375 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.289515 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.304283 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:07.307960 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:07.311618 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.317014 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.319406 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.328366 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.330499 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.334586 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.348009 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.419097 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.421996 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.426267 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.428853 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.442083 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.457886 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.467252 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.502528 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.505712 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.508143 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.510664 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.511915 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.520119 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.554987 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:07.574431 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:07.585796 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.591174 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.592821 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.595759 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.600497 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.625677 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.626458 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:07.664330 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240503 08:01:07.698285 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.700134 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.705154 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:07.706344 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240503 08:01:07.712911 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.729364 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.735880 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:07.743628 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.748498 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:07.771731 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.773190 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.785889 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.796360 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.796855 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.805846 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.811211 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.830324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.848637 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.851227 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.855669 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.857743 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.860442 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:07.883073 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.891460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.962473 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:07.963299 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:07.964291 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:07.980312 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:07.983783 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:07.985383 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:07.985581 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.011060 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.079006 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.081259 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.088598 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.088793 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.092975 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.101341 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.101970 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.119609 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:08.133703 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:08.134619 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.134974 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.137086 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:08.147770 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:08.169623 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:08.176312 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:08.177291 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.178086 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.217023 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.223424 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.231240 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.232952 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.235663 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.235782 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.243218 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.244340 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.251336 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.264503 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.268741 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.281387 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:08.291447 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:08.292833 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.313087 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:08.380138 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.422745 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.424420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.433202 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.441440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.457093 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.463228 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.465544 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.483197 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.485846 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.489162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.490667 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.499774 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.504320 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.505390 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.522205 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.528369 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.529273 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.539871 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.540076 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.543361 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.594113 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:08.604396 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.614411 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.650316 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:08.660586 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.660992 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.665040 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.677605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:08.680672 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.682837 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.687085 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.692030 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:08.715209 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.749327 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:08.758419 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:08.761417 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.769002 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.769801 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.782986 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:08.784512 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.791143 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.816334 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.816336 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.817526 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.831322 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.833938 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.848171 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.850066 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.852053 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.862844 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.883248 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:08.883414 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.889025 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.889665 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:08.889819 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:08.898692 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:08.934970 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:08.943289 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:08.972875 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:08.981598 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:08.997697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.000537 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.008626 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.012038 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.014883 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:09.018628 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:09.027081 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:09.140705 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.145485 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.161801 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.163662 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.165459 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.175447 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.184619 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.189316 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.189742 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.191254 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.212780 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.221954 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.223841 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:09.228502 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:09.233724 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.237790 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.238806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.251224 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.254968 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.257814 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.258749 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.264101 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:09.271901 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:09.273018 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.276890 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.280936 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.287019 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.305794 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.331424 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.361248 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:09.384974 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:09.444036 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.445263 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.453074 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.462486 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.466434 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.469714 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.478629 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.479179 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.483086 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.490239 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.490702 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.495607 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.512351 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.523464 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:09.557861 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:09.561113 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:09.629925 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.632369 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.640458 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.642972 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.646653 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.661510 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.669418 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.728611 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.736532 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.739514 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.740429 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.755739 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.757268 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.762948 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.780282 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.781231 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.786605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.787072 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.799448 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.832908 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:09.839813 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.849967 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.857882 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:09.860093 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:09.867136 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:01:09.869045 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.873929 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:09.875247 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.885569 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.968402 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:09.970415 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:09.975080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:09.975338 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:09.996999 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:09.997671 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.008684 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.012614 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:10.023209 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:10.087327 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.089049 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.094704 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.098068 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.103161 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.106667 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.107931 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.128221 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:10.129755 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:10.131820 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.133036 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.138103 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.142167 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.156809 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.159615 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.165190 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.171386 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.173611 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.174475 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.176482 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.201225 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.208511 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.334075 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.346050 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.348838 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.349822 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.360896 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.367363 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.377130 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:10.407013 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:10.410562 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.415669 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.422452 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.432345 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.436830 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.442847 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:10.457939 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:10.471498 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:10.471805 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.506484 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.506511 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.506650 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.507853 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.510428 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.532577 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.541704 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.551080 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.568900 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.571892 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.574368 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.574862 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.604408 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.608263 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.649524 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.649765 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.650060 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.651315 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.651737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.655364 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.662638 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.751071 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.758195 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.773599 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.775416 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.793985 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.796792 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.810159 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.855571 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.856170 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.861117 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.861650 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.875607 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:10.876056 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.884613 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:10.896288 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.898514 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:10.928674 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:10.937165 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:10.938338 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:10.941479 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:10.949723 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.960189 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:10.989974 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:11.004047 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:11.007781 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:11.011905 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.033125 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.033188 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.034021 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.034265 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.034344 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.055886 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.082026 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.082708 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.082800 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.101820 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.116617 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.125217 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.125443 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:11.132611 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:11.198398 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.210486 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.210763 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.218979 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.234525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.236328 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.255873 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.265133 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.269870 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:11.273116 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:11.273460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.274322 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.279836 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.284080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.291843 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:11.306571 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.311313 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:11.412335 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.413590 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.415696 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.416425 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.426257 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.432701 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.447117 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.459584 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.464857 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.482441 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.485054 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.485217 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:11.499980 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.502664 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:11.502842 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.511457 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:11.514422 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.523092 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.558192 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:11.561161 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:11.566965 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.570905 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.571163 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.571643 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.573622 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.576820 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.586297 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.596331 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.601893 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.614682 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.614702 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.619396 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.628707 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.729557 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.743968 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.745747 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.747351 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.757555 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.758150 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.758527 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.774688 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.774727 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.781416 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.792608 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.801185 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.801360 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.809587 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.875475 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:11.879448 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:11.881630 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:11.882835 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:11.884145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:11.884796 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:11.885318 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:11.928939 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.936590 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:01:11.975301 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.991056 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:11.993878 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.003306 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.007961 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.008899 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.009557 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.018163 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:12.024899 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:12.041280 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.051075 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.062028 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.062783 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.071354 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.089962 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.094606 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.108999 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.168488 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.169083 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.170627 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.172011 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.181875 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.186569 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.213435 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.291718 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.294353 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.300611 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.301880 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.306545 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.308579 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.333132 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.360510 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.363442 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.369984 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.370465 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.372480 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.383700 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.385041 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.388732 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.403592 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.414479 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.417012 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:12.422648 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 08:01:12.423282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:12.434837 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:01:12.435534 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:12.439088 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.455233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.501531 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.510524 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.514056 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.514423 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.517385 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.530414 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.533819 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:12.550422 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:12.606837 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.610196 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.616015 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:12.623463 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:12.624533 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.629475 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.641794 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.643137 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:12.668434 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:12.669196 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:12.688692 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.690598 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.691455 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.695538 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.696002 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.711855 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.718670 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.793859 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.802901 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.809684 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.823853 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.825001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.828210 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.828547 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.830451 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.832909 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.844643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.846959 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.857966 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.874447 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:12.878612 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:12.913627 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.919248 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:12.922500 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:12.922902 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:12.925520 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:12.928876 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:12.942844 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:12.946751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:12.981513 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:01:13.079519 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.083057 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.085832 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.090855 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.110131 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.121826 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.123951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.145094 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.147904 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.149621 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:13.158320 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:13.168978 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.174285 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.177186 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.198885 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.206624 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.219060 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.228991 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.235138 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.238338 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.250840 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.251269 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.251457 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.260001 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.286567 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.288686 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.291632 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.315078 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.319175 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:13.349665 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:13.427897 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.428227 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.432962 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.436641 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.437237 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:13.438787 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:13.439671 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:13.440027 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:13.441766 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.462550 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.465787 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:13.472632 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:13.491164 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.495368 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.509398 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.533740 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.548593 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.553606 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.559078 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.565912 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.568059 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.580334 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.584507 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.592371 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.594349 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.599068 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.619812 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:13.644587 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:13.664304 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.673725 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.674907 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.684834 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:13.694437 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:13.720049 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:13.780787 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:13.787022 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:13.787827 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.788256 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:13.789098 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.789461 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.792326 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.805598 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.811854 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.814616 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.844795 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.865317 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.874243 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:13.879456 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.890018 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.897293 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.914963 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:13.933885 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:13.954042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:13.957756 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.959781 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:13.960309 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:13.962080 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:13.967769 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:13.975195 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:13.993952 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:14.014526 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:14.022358 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:14.085762 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.095654 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.095743 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.095782 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.095999 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.098611 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.104873 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.120887 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.123636 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.126550 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.138473 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.139186 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.155143 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.161761 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:14.214076 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:14.228808 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.235868 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.239302 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.240906 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.248726 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.248786 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.258085 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.355597 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.357718 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.374210 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.387229 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.389534 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.401890 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.405860 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.408718 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.423297 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.429944 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.448516 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.453969 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.457818 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.469463 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:14.495103 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:14.497838 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.501402 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.509110 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.514222 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:14.523072 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:14.523146 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:14.531630 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.535418 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.539598 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.546581 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.549219 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.558681 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.561722 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.582399 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.594429 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.594833 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:14.614699 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:14.623140 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:14.679800 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.683486 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.683868 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.693279 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.694952 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.703981 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.728430 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.782317 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.784258 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.784597 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.788511 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.790585 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.799101 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.799612 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.812027 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.825537 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.828660 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.828841 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.850109 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.855358 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.865005 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:14.923877 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:14.941099 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:14.941921 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:14.943512 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:14.969470 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:14.975275 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:14.976802 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:14.991820 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.008117 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.019631 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.021637 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.022184 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.023527 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:15.030779 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:15.043893 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:15.067226 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:15.067538 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:15.117976 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:15.144423 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.147672 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.153375 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.155761 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.166986 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.169580 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.184473 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.184603 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.188338 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.189806 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.222272 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.225768 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.234813 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.235872 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.244170 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.250833 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.251278 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.266155 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.298518 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.299759 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.314234 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.411980 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.412297 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.422433 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:15.427016 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:15.428233 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.430663 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.431041 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.435096 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:15.442957 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:15.449872 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.468607 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.478056 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.478933 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.480715 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.488389 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.493731 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.555176 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.564543 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.568038 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.576781 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.576823 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.579686 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.585897 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:15.647078 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:15.693018 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.693405 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.694322 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.697542 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.706107 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.722569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.726258 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.756353 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.771013 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.777261 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.782014 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.783351 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:15.784204 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:15.791240 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.794298 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.805806 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.818150 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:15.819026 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.826946 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:15.835342 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.839597 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.850306 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:15.854189 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:15.857010 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.859084 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:15.868456 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:15.871766 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:15.876216 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.880824 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:15.882320 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:15.905792 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:15.941265 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.021284 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.024484 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.030575 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.040503 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.056797 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.062283 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.081992 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.104348 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.106060 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:16.107443 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:16.109412 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.112951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.119007 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.156132 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.160145 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:16.197409 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:16.200023 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.218329 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.218366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.221000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.221261 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.221755 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.237890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:16.250749 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:16.263558 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.277540 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.278793 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.288507 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.288683 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.293331 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.293596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:16.299356 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.306294 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:16.362111 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.362149 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.363097 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.370966 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.380887 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.381901 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.387760 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.456248 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.456646 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.464017 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.471822 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.475086 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.491988 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.505297 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.547922 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:16.548507 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:16.555941 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.567406 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.571957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.587705 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.591972 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.609501 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.611799 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.620635 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.636027 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.638577 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.642141 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.645931 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.657699 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:16.670408 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.699957 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.703905 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:16.769398 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.769995 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.771647 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.791493 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.796114 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:16.796840 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:16.815635 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.838065 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.848186 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.856323 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.861061 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:16.868880 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:16.869325 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:16.872572 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 08:01:16.872660 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.873313 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.873785 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.877628 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.888818 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.896741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.903255 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.903961 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:16.925374 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:16.927309 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:16.940552 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.954186 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.960171 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:16.962239 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:16.968035 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:16.976096 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:01:16.982861 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:16.999720 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:17.087003 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:17.105101 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.111467 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.112463 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.114450 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.125900 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.131588 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.167932 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.169924 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.178797 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.181535 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.184633 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.184962 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.185196 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.187057 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.216795 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.218950 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.227957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.237881 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.240937 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:17.252144 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:17.253183 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:17.255548 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.260217 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.408282 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.409081 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.413456 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.419797 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.426960 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.431623 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.445315 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.457906 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.459890 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.475482 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.490831 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.492045 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.496704 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.514849 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.515501 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.521497 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.530131 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.532716 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.548238 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.562819 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.567478 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.568603 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.585697 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.589572 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.592272 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:17.595228 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:01:17.601137 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:17.601506 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.602708 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.630021 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:17.703814 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:17.724752 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.726701 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.728344 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:17.738657 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:17.739267 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.751375 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.751433 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.755086 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.803241 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.810364 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.812181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:17.812871 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.819386 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.826539 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.840008 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.840560 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.845979 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:17.853894 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:17.864604 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.872684 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:17.874366 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.889294 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.904322 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.961961 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:17.962323 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:17.974148 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:17.977283 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:17.980688 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.000545 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:18.001353 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.002995 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.033951 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.042523 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.047473 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.054215 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.057734 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.060142 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:18.061314 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:18.080577 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.123554 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.124410 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.132716 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.139638 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:18.151453 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:01:18.152568 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.153770 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.155023 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.162652 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.223634 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.231074 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.233245 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.234736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.244402 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:18.253396 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:18.271292 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.278306 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.345681 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.361763 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.370011 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.370976 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.374332 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.388109 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.390123 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:18.390388 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.390836 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.404865 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.405565 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.406260 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.412323 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.425696 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.439463 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:18.440212 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:18.494400 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:18.538455 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.540627 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.545038 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.553433 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.553722 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.554003 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.557356 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.558795 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.559048 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.581244 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.591686 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.599084 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.607745 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.620718 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.623878 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.643333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:18.646970 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.648428 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.648921 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.652066 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.670567 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.694984 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:18.701020 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.753698 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.758523 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.765125 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:18.766795 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:18.777756 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.779584 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.801441 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.802976 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:18.816165 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:18.858625 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.860090 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.862993 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.866796 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:18.878479 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:18.882539 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.887132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.888880 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.890148 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.894368 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:18.904563 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:18.914878 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:18.924029 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:18.930022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:18.930832 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:18.987159 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.990262 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:19.028751 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:19.042280 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.045315 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.057178 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.057541 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.069514 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.072198 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.085044 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.091733 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.093789 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.101691 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.110003 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.110627 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:19.118422 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:19.131868 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.149899 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.156977 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.160080 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.160712 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.173405 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.187384 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.201159 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:19.207875 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:19.215261 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:19.322487 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:19.323182 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:19.331493 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:19.342756 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 08:01:19.343194 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.350979 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.352197 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.353364 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.353574 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.358000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.367085 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.385922 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.390370 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.396884 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.413477 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.414969 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.417754 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:19.462635 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:19.465629 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.473150 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.484889 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.496659 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.496946 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.500101 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.508674 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.524173 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.527534 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.535902 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.537187 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.549790 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.562769 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.573628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.622835 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.629880 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.631742 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.648474 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.661837 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.664425 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.678346 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:19.685375 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:19.778692 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:19.804000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.817675 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.821426 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.822258 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.822430 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.833050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.835088 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:19.852521 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.880125 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.885251 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.889657 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:19.890755 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:19.894692 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.896539 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:19.896867 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:19.897661 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:19.906013 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:19.919287 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.955219 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:19.956223 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.981633 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:19.986059 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.000818 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.007474 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.011971 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.018443 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.022967 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.032873 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.033495 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.056628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.130338 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.135305 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.135375 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.139469 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.172430 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.176105 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.178278 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.182252 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.185184 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:20.191143 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:20.207748 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.210876 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.216464 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.224495 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.226616 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.341933 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:20.344584 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:20.354954 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.355386 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.386068 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.392987 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.399773 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:20.400336 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:20.405014 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.405613 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.413642 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.424799 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.425453 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.436784 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.455613 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.466393 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.478580 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.494700 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.500638 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.503012 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.503026 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.504703 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.504812 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.548214 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.548821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.554503 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.563304 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.570026 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.595440 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.598677 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.666298 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.667505 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.668848 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.670720 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.680330 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.686386 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.687194 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.770408 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.780562 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.781726 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.784934 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.791421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:20.793272 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:20.795641 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.810006 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:20.815711 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:20.825277 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.827575 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:20.847887 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.849711 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.853755 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:20.855080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.856225 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:20.861865 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:20.883127 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:20.884630 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.886354 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:20.888341 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:20.921186 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.922278 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:20.929736 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:20.954962 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:20.959025 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.065994 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.067570 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.073292 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.082295 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.084739 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.092484 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.093510 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.093658 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.107107 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.108845 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.117801 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.121042 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.135303 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.140708 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.146483 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.159816 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.163722 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.168761 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.171656 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.173964 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.182065 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:21.255589 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:21.287120 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.295207 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.313482 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.315938 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.325698 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.326336 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.326717 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.333240 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.366514 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:21.374492 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.377796 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.378000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.386173 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.411412 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.414887 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:21.422863 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.426214 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.426285 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.456552 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.456771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:21.456846 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:21.461010 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.465988 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.476994 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.482837 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.503742 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.617239 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.629645 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:21.629830 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.630628 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.644223 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.650635 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.662657 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.666698 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.679898 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:21.686800 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.686861 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.687769 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.697602 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.702386 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.705088 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.707590 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.733150 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.736217 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.740002 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:21.752626 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.767102 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.770946 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.775270 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:21.785473 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.788049 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.791078 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:21.808926 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.825253 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:21.830729 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.844081 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.846321 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:21.848186 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:21.850548 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.860235 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:21.863569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:21.869028 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:21.870081 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:21.878976 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:21.892117 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:21.892191 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:21.905869 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:21.941648 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:22.014289 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.016276 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.024048 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.025554 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.026070 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.027617 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.055097 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.063974 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.069484 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.072614 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.080528 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.082098 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.092583 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.109220 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:22.131083 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:01:22.229707 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.235347 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.239341 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.242888 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.259513 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.260936 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:22.269944 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:22.298418 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.299705 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.313469 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.326004 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.326346 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.328172 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.341197 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.358656 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:22.359154 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:22.374732 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.375100 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.400687 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.407953 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.413641 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.414297 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.415364 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.421857 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.422456 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.431255 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.443001 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.443678 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.456141 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.465474 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.479156 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.489581 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.499883 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.509151 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.510613 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.523516 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.534942 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:22.584745 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 08:01:22.621886 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.623869 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.632458 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.632455 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.641628 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.645453 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.651723 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.684074 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.687021 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.693274 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.696683 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.705530 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.715835 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.730859 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:22.761986 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:01:22.762742 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:22.829087 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.830565 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.835381 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:22.835485 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.841187 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.841300 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.842690 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.846309 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.861483 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.898458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.916347 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:22.919734 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:22.933161 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:22.938534 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.943104 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:22.943104 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:22.950765 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:22.953461 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:22.972493 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:22.978858 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:22.988552 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.000928 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.009371 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.028254 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.035059 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.041514 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.049505 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.055138 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.056314 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.070578 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:23.114403 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:23.138007 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.142469 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.143136 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.144394 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.155462 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.170516 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.189000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.193123 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.194463 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.196012 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.207613 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.220978 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.223013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.226084 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:23.237922 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.287968 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:23.288599 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:23.383957 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:23.386515 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:23.389631 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.396780 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.404609 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.417210 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.431576 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.431631 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.466789 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.473925 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.477480 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.485538 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:23.495044 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:23.500031 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.509102 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.510058 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.520699 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.523326 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.523917 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.524509 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.534682 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.537266 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.539333 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.543354 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.548456 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.550827 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.564745 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.567498 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.578169 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.582487 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.586426 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.594249 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.601712 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.603224 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.603546 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:23.620268 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:23.630165 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.647871 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:23.650164 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:23.724639 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:23.767040 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.769276 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.769874 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.770046 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:23.771322 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.785781 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.799728 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.799973 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.802254 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.822785 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.826704 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.827546 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:23.828611 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:23.835805 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:23.836155 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:23.853237 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:23.904563 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:23.928941 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:23.944954 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:23.956225 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:23.979413 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:23.980875 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:23.981572 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.000293 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.016209 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.020792 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.025071 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.040045 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.045457 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:24.045461 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:24.047339 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:24.061048 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:24.064100 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:24.069012 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.070632 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.093322 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.124785 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:24.134995 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:24.139748 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.146589 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.157150 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.160327 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.167173 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.173842 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.174105 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.180970 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.190742 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.210356 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.211066 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.226212 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.226473 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.232461 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:24.233474 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:24.247918 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.250417 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.255218 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.263664 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.283241 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:24.359474 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:24.365831 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.373423 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.380999 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.382901 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.392649 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.404106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.411305 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:24.411489 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:24.439424 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.443044 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.448107 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.452543 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.474049 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.497431 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.499094 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.512560 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.513581 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.532534 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.534411 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.537737 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.547621 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.573426 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:24.583079 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:24.624439 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.624902 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.635692 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.641165 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.641330 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:24.642760 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:24.645287 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.646818 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.653264 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.654467 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.666201 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:24.668653 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:24.669684 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.676517 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.677923 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.709864 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.773545 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.781016 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.785274 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.794008 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.803395 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.804589 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.807431 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.900622 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.904045 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:24.904492 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.904733 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.906509 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:24.910583 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.927281 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:24.927850 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.937201 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.937539 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.944490 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:24.960037 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:24.962908 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:24.966217 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:25.021957 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:25.044241 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:25.056439 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 08:01:25.057191 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.058606 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:25.064558 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:25.068387 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.069186 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:25.072410 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:25.079373 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.080053 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.098116 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.109582 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.111670 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.122469 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.124588 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.133571 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.139353 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.208164 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.209326 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.221087 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.223850 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.232698 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.243093 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.267074 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.287541 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.289726 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.293315 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.303786 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.304283 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.307864 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.354357 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.373703 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.380098 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.380642 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.389175 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.392005 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.414036 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.435400 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.500190 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.501713 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.511989 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.517277 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.517352 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.518095 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.521061 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.523692 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.526031 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.527333 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.535436 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.560870 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.573089 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.588189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:25.595528 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:25.667002 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.672174 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.678499 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.691496 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.696759 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.696993 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.698109 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.706413 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.706723 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.707413 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.707777 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.711519 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.757910 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:25.758002 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:25.758492 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:25.779189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.814429 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.817979 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.824688 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.848948 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.856209 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.857013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.861521 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:25.899209 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:25.901288 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.902688 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.902981 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.925005 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.925164 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:25.935701 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:25.966953 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:25.975239 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:25.980199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:25.983779 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:25.996932 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.017210 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.017918 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.019299 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:26.057870 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:26.125665 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.137063 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.139567 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.149720 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.155050 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.164749 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.176353 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.179845 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.187870 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.195297 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.210083 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:26.211400 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:26.215453 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.218240 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.232213 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.232650 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.240679 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:26.273669 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:26.275475 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.277267 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.277631 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.277846 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:26.290796 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:26.307405 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.314304 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.316605 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.320667 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.322593 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.332291 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.338608 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:26.345412 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.351523 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:26.355959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.398640 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.399284 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.400449 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.407994 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.415143 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.421358 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.425388 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:26.444010 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.473470 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:26.532506 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.535995 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.538825 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.549974 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.551378 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.556247 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.566213 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.568679 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:26.572892 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:26.573386 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.580871 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.597165 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.601022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.605752 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.619691 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:26.651542 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.735072 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.749207 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:26.780587 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.790977 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.793040 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.794123 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.808795 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.811962 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.813287 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.814808 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.816306 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.817791 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.849145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.850733 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.855846 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:26.863165 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:26.873886 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.874486 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.878379 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.883507 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.886094 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.887841 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.900094 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.912696 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:26.927847 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:26.927860 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.928145 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.930310 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:26.932998 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.948696 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.954811 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.963924 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.969671 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.974261 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:26.984434 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:26.988579 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:26.989488 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:27.011253 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:27.019505 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.147518 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.148550 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.160091 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.160573 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.164327 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.175462 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:27.175694 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:27.188107 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.193051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.216746 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.222465 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.222740 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.243577 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.246946 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.252760 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.299955 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.304255 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.304247 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.309341 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:27.331022 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:01:27.331866 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:27.336557 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:27.343158 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.343670 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:27.349714 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:27.412226 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.415333 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.419826 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.428206 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.441666 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.448776 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.455657 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.456037 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.457906 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.468459 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.481571 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.491065 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.506331 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.507597 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:27.529579 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 08:01:27.587433 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.592581 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.594769 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.607930 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.614774 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.615304 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.629660 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.632587 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.639299 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.654415 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.657609 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.658476 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.664006 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.670883 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:27.701900 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:27.768433 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.771054 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.791810 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.794593 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.798105 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.822640 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.826861 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.875891 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.885865 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.886371 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:27.892714 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:01:27.893333 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:27.902736 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.907665 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:27.926467 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.944171 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.958685 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:27.960412 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.963435 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:27.966464 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:27.971396 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.974117 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:27.975435 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:27.979838 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.004128 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.011858 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.013499 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.042826 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.043262 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.054127 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.088039 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.091300 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.106472 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.106623 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.121438 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.122673 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.131769 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.208917 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.216053 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:28.224154 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:28.225116 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.233915 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.238018 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.240692 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:28.258280 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:28.259460 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.259851 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.264277 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.279834 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.293458 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.299063 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.310256 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.311080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:28.311614 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:28.314813 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.321660 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.347030 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.360080 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.368746 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.376533 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.377635 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.478135 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.491883 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.494947 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:28.502661 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:28.503554 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.503971 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:28.511950 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.519556 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.524080 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.525923 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.527783 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.547864 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.549491 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.550717 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.560936 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.568511 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.575374 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.578891 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.580905 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.592901 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.603138 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.624503 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.631551 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.665261 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.671737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:28.679456 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:28.680694 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.685153 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.685751 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.692028 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:28.699489 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:28.706507 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.709815 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.780930 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.784137 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:28.784164 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:28.788367 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.788481 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.788602 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.789535 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.809288 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:28.848747 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:28.889544 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:28.890187 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.890391 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:28.891007 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.897382 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.916289 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.922053 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:28.947088 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:28.960767 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:28.981164 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:28.981544 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.005779 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:29.007923 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:29.008078 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.018813 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.022811 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:29.065232 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:29.068339 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.076375 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.077683 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.079180 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.085126 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.085738 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.090451 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.097679 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:29.103781 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:29.104463 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.110240 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.116011 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.121080 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.124536 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.135104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:29.137167 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.140218 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.154911 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.219782 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.228818 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.242554 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.243969 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.259153 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.288051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.289906 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.294013 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.303902 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.304127 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.304934 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.316860 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.317533 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.332041 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.334750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.335675 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.338829 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.349704 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.373824 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.374081 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:29.393064 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.395660 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:29.417994 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:29.471139 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.494879 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.496977 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.497488 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.504608 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.508666 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.509235 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:29.521936 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:29.528915 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.530004 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.530607 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.534819 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.557005 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.561196 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:29.572376 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.579492 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.583763 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:29.585702 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:29.621922 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.624198 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.627890 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.629076 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.643194 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.646759 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.659556 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.756621 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.756816 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.758180 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.759835 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.771999 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.773219 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.789222 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.790864 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:29.796792 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:29.798612 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.800350 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.805706 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.811311 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.813015 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:29.817407 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.825019 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:29.829006 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:29.829278 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:29.838765 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:29.856218 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:29.860909 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:29.871073 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:29.872110 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:29.900851 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:30.005339 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.007052 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.007946 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.015102 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.015430 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.035475 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.037487 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.043421 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.050246 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.051033 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.054982 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.064199 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.079173 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.106856 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.124742 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.130815 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.143468 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.157534 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.165751 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.172158 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.180383 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:30.224186 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:30.265081 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.275087 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.276223 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.276337 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.286269 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.298797 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.302232 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.307237 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.315999 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.323074 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.332576 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.338022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.347055 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.372520 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.398901 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.401269 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.402858 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.420306 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.421582 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.422885 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.448984 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:30.452735 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:30.488127 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.494709 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.494801 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.501111 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.505349 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.513989 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.524274 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:30.531877 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:30.582559 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.584200 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.594837 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.598038 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.613674 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.616514 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.621906 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.626734 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.635105 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.660882 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:30.669939 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:30.671757 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.672832 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.677942 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.693763 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.765527 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.769781 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.791057 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.797518 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.801918 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.812361 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.812736 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.825073 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.828845 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:30.832101 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:30.836319 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:30.836544 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:30.841653 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:30.851298 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.854068 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.955730 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:30.979364 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:30.983418 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.999109 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:30.999159 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.022511 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:31.024075 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:31.032276 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.097648 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.102538 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.116444 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.117995 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.125938 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.145010 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.157786 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.169087 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.177583 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.183962 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.191082 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.191614 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:31.205013 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.213963 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.224134 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:31.254079 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.255743 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.260088 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.261200 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.274812 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.277401 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.283421 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.294384 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:31.304297 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.343292 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.354966 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.359839 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.377986 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.381310 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:31.382727 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:31.387820 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.388223 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.390239 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:31.463124 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.474792 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:31.475051 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.486577 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.494222 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.499918 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.500650 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.517951 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.518126 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.518496 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.519419 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.523550 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:31.531597 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:31.538499 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:31.540040 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.548851 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.559914 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.577924 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.593891 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.606840 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.609011 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.612790 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.616114 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:31.622913 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.635061 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.635347 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.636270 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.636966 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.636969 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.658871 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.663681 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.672505 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:31.706530 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.709914 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.781129 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.781828 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:31.797880 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:31.798645 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.798816 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.808068 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.821081 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.821162 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.851274 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:31.852173 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:31.852201 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:31.854519 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:31.858774 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:31.859215 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:31.867767 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:31.876080 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.928530 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.946789 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:32.005527 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.015329 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.019945 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.027801 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.029161 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:32.038250 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:32.045918 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.068254 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.072749 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.081995 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.094779 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.095103 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.098449 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.110002 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.115022 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.124265 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.124964 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.145304 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.151942 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.161895 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.174525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.175292 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.178958 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.197752 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.214581 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.217842 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.218171 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.221102 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.249013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.307803 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.310082 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.313241 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.324623 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.327503 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.351034 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:32.351665 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:01:32.358487 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 08:01:32.361143 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.388980 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.390769 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.392163 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.397272 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.398957 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:32.408175 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:32.409061 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.419948 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.471675 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.477445 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.481238 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.486553 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.490826 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.496273 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.507897 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:32.525795 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:32.591967 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.597462 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.607024 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.608331 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.621496 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.628683 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.632484 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.633308 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.635716 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.638092 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.645228 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.651978 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.653301 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.654888 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.658788 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.669103 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.682361 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.684727 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.700136 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.704288 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.716318 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.798967 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.804236 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.814747 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.816025 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.816068 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.837648 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.837874 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.888381 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.897260 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.897827 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.919142 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.923596 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.929201 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.940241 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:32.962210 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:32.962981 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:32.969956 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:32.973028 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:32.973734 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.982590 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:32.984323 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:33.008960 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 08:01:33.009625 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:33.028597 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:33.046063 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.048406 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.051379 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.055146 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.073987 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.077052 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.124594 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.164381 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.167506 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.175873 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.178237 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.183082 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.192039 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.194630 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.234815 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.242594 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.243335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.251470 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.258723 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.267879 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.270459 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:33.288869 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:33.330853 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.334861 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.338668 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.342712 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.345726 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.347560 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.363322 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.374979 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.381844 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.388806 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.393069 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.393675 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.398620 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.399076 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.464728 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:33.473160 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:33.479837 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.484979 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.485539 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:33.486330 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:01:33.495186 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:33.507503 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:33.507767 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.516669 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.582906 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.593199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.593937 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.604904 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.608270 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.608913 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.621578 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.633008 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.637869 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.652384 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.654851 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.667011 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.669281 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.670379 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.681840 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.683831 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.690673 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.695886 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.704119 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:33.704403 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:33.716847 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.720669 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:33.737498 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:33.738095 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.826452 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:33.835481 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.846513 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.859768 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.860944 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.865721 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.876893 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.880667 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.886049 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:33.886370 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.892778 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:33.898983 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:33.902766 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.902791 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.904940 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.909492 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.939065 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:33.950165 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:33.951413 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:33.963488 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:33.969554 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:33.979432 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:33.996999 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.003585 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.015312 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:34.041831 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:34.042605 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.099315 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.107084 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.107264 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.114867 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:34.127722 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.139909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.141655 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.155874 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.161188 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.164909 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.167801 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.170826 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.173390 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.180179 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.182139 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.188329 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:34.205591 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.213883 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.227145 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.227598 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.235730 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.243759 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.244056 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.277829 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.321663 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:34.324385 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.342350 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.351817 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.352097 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.353682 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.358014 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.359323 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.403339 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.405534 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.407122 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:34.407492 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:34.416954 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.429239 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.438369 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.454028 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:34.456584 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.477735 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:34.511930 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.512423 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.519083 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.526950 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.531535 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.531579 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.539767 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.546449 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.553540 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:34.554517 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:34.569435 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.571280 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.583070 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.583794 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:34.586473 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:34.586963 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.661487 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.668323 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.673236 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.675714 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.692458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.695977 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.722316 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:34.747296 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:34.756623 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.763851 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.775568 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.776397 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.779172 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.788192 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.792429 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.802644 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.803958 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.804945 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.818573 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.818964 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.826314 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.833313 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.930121 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:34.935818 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:34.937368 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:34.939168 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:34.942906 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:34.952513 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:34.973554 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:34.980866 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:35.035532 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.050841 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.079885 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:35.088898 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:35.089088 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.089408 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.097978 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.100001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.105300 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.127107 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.127110 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.140514 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.145884 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.148083 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.149088 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.155653 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.156523 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.167057 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.170624 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.197652 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.213043 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:35.223239 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 08:01:35.226402 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.269160 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.272018 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.277817 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.282351 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.282521 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.288797 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.327358 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.372350 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.373404 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.375833 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.383069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.393090 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.401649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.403903 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.456394 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.456596 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.458482 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.459645 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.462020 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.469843 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.473706 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.477319 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.483069 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.485142 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.500245 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.511893 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.526861 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.538913 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:35.592850 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:35.593216 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:35.626766 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.628275 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.631098 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.635152 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.655485 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.674852 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.675482 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.708854 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.732249 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.738665 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.739279 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.760885 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.763159 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.769121 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.778985 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.781339 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:35.787887 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:35.788707 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.793640 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.813874 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.843817 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.847985 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.858536 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.861119 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.873219 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.878223 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:35.883227 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:35.905406 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:35.908438 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.910540 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.912679 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:35.919055 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:35.927441 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.932149 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:35.952924 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:35.955569 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:35.962738 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.007807 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:36.050701 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.053016 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.062788 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.068912 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.074456 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.085465 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.089053 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.089630 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.089862 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.100973 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:36.102221 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:36.111407 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.115844 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.116636 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.117084 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:36.126014 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.240662 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.243007 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.248473 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.256548 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.272534 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.272570 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.281334 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.291265 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.294270 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:36.294767 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.305994 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.306744 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.351603 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.352456 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.355978 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.364331 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.398846 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.399827 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:36.400741 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.410208 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.417229 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.421154 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.422487 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.425899 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.465107 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.506870 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.513854 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.522266 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:36.523491 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:36.525110 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:36.528394 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:36.532294 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.535598 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.544047 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.549820 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.573992 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:36.574194 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.578642 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.607069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.613546 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.614687 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.618688 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.621189 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.630676 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.631489 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.638410 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.651605 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.657827 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.659773 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.667397 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.668849 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.668963 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.669740 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.672160 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.704699 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.715377 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.716620 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.792042 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:36.799810 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:36.883512 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.888453 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.898789 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.900413 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:36.903147 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:36.903594 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.920763 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.925356 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.927052 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.928098 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:36.941596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:36.942152 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.954262 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.956857 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:36.959764 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:36.969079 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:36.971765 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:36.982080 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.001811 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.009620 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.014966 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.017606 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.022367 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.030388 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.032106 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.035686 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:37.038093 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:37.079326 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:37.081245 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:37.092430 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.104185 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:37.141463 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:37.171875 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.172123 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.175146 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.180414 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.190454 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.191943 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.208958 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.243870 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.246019 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.249621 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.255100 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.258194 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.260931 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.282361 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:37.283221 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:37.290220 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:37.376534 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 08:01:37.401348 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.402232 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.402349 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.403419 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.418622 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.421479 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.422780 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.464002 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.465764 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.472138 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.476744 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.478781 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.496946 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.509682 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.548041 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.549482 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.553702 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.555382 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.560825 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.565315 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.576387 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.586302 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.595923 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.596069 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.596632 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.599910 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.603120 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:37.606173 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:01:37.610941 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:37.663792 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:37.705111 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.709390 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.710008 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.724913 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.748889 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.750659 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.755057 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.790666 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.794154 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.797890 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.812955 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.817617 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.845482 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.863384 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.866814 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.873929 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.880666 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.884930 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.887720 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:37.888479 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.891366 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:37.902601 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:37.913906 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:37.915902 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.917286 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:37.923409 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:37.933892 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:37.952478 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:37.957255 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.019945 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.020629 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.028764 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.030162 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.030635 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.037145 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.047531 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.120411 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.124653 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.125140 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.151681 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.166867 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.171044 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.180807 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.186729 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.186985 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.187697 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.189532 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.217569 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.219350 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.243582 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:38.261415 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:01:38.262040 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:38.324115 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.332933 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.335382 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.351066 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.367646 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.367777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.368005 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:38.375826 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:38.394521 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.415023 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:38.423785 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:38.438364 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.440014 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.451123 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.462354 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.462615 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.462631 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.464679 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.472009 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.481832 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.482542 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.497449 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.498118 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.508957 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.517416 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.522087 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.530571 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.543877 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.547789 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:38.560128 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:38.565588 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:38.685402 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:38.697556 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.702684 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.705408 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.709800 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.712414 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.722196 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.732787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.736502 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.740721 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.747536 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.756556 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.764866 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:38.771939 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.785089 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.799058 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.803354 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.812947 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.822834 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:38.826395 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.834951 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.839316 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:38.845256 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:38.853099 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:38.859102 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:38.864696 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.867076 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:38.871083 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:38.871228 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:38.893815 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:38.907799 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:38.924487 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.929494 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:38.949779 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.980947 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:38.999745 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.000185 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.003139 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.012616 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.014102 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:39.021834 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:01:39.022728 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.024502 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.040836 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.047528 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.054610 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.060886 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.061033 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.074355 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.100690 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.152323 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.161217 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.181385 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.183871 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.189062 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.190096 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.211613 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.263257 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.264611 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:39.270432 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.292068 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:39.293879 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.297859 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.308754 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:39.320113 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.325906 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.329993 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.333223 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.336967 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.352088 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:39.357933 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.358670 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.360949 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.367446 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.374969 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.395722 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.398666 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.398836 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:39.407802 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:39.421921 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.423561 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.427623 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.455757 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:39.462688 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:39.465574 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:39.490970 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:39.502377 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.511324 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.513705 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.523828 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.538810 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.538887 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.546991 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.582170 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.585508 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.589049 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.591775 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.596815 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.603545 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.617563 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.666421 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.667801 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.671280 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.673107 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.675649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.682698 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.693534 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:39.696368 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.764034 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.777498 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.780105 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.781272 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:39.784453 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:39.787077 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.798203 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:39.799619 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:39.820792 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:39.827548 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:39.840477 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.842100 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.845345 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:39.848414 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.860836 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.862942 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.865028 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.908380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:39.918453 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:39.919736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.920604 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:39.934573 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:39.935595 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:39.946789 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:40.049041 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:40.073747 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.074033 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.076808 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.085650 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.095301 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.110220 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.115602 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.125247 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.126827 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.134861 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.139416 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.152829 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.162149 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:40.168591 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 08:01:40.172366 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.184211 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.192044 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.192972 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.196200 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.199090 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:40.211786 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:40.220144 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.221444 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.237337 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.240546 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.251111 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.252687 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.253159 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.264711 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.307251 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.371949 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.376878 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.379633 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.383371 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.389300 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.400501 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.401368 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:40.463232 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:40.478416 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.481359 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.482973 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.491467 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.493449 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.494055 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.498284 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.509303 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.518827 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.534272 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.535929 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.545127 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.555749 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.578320 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.645740 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.659525 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.660923 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.663266 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.682734 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.699872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.705248 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.717475 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.724730 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.736337 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.736987 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.746085 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.756449 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.764817 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.770810 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:40.782406 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:40.788913 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.791146 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.791289 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.811393 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:40.823740 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:40.827741 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.829033 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:40.852679 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:40.882287 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:40.882570 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:40.891989 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:40.892604 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:40.893239 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.894338 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:40.896981 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:40.923939 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.986757 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:40.988902 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:40.995198 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.003293 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.006170 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.020650 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.022529 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.105785 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:41.115832 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.117846 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.120900 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.127301 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.131022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.137084 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.138705 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.163705 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.165616 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.176466 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.185750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.187511 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.197517 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.214471 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.298022 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.301627 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.302894 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:41.314574 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.316985 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.323894 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.325428 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.329449 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.344676 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:41.359719 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:41.362891 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.364293 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:41.373468 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.376230 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.384703 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:41.390439 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:41.395118 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.403333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:41.407029 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.452575 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.472045 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.478170 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.481740 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.486429 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.500490 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.503139 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.520639 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.532649 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.533064 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.535002 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.540431 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.540477 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.544708 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.553504 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.555394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.557132 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:41.563359 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.585850 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.603745 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.606588 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.613418 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.729116 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.743125 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.744484 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.754372 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:41.757045 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:41.767941 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.776744 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.778437 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.781702 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.782372 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.786492 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.786782 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.815223 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.817106 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.836330 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.843186 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.849382 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.868575 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:41.872350 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:41.878844 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.879633 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:41.885941 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.886737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:41.887830 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.892719 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:41.906718 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:41.910957 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:41.910965 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:41.911516 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:41.914134 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:41.921782 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:41.945509 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.966717 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.971596 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:42.033957 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.037061 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.044442 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.047801 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.051838 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.057711 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.064548 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:42.121726 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:42.128099 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.136130 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.139425 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.144477 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.163831 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.167078 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.169523 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.173003 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.189252 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.205994 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.215598 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.223078 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.226262 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.232465 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:42.256158 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:42.293499 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.294517 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:42.296114 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:42.299343 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.305366 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.316994 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.344446 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:42.349221 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:42.365449 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.381877 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.388445 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.388445 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.389127 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.392912 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.409392 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.421023 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:42.435384 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:42.435554 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:01:42.493206 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.496074 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.502575 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.506356 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.513255 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.527576 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.528780 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.560946 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.562445 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.568846 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.585798 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.591691 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.600275 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.605965 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.633185 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.639777 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.643110 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.644042 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:42.653395 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:42.653739 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.678968 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.688212 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.701686 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.705312 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.705312 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.714782 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.721715 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:42.732455 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.739645 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.821002 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.824954 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.830713 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.832578 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.843811 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:42.845934 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.855724 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:42.857023 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:42.917241 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:42.929308 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:42.956218 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:42.966450 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:42.966827 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:42.971179 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:42.976388 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.000824 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.003916 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.006702 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.013473 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.043838 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.047672 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.050849 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.056195 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.065049 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.071904 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.083314 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.093343 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.093834 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.101979 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.112885 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.196055 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.214488 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.215396 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.232547 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.241315 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.243065 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.243731 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.244036 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.249179 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.249423 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.256698 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.264956 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.267113 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.293994 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:43.361694 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:01:43.362308 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:43.364931 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:43.366552 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.367684 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.368914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.377360 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.381700 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.402287 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.415771 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:43.439200 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:43.476821 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.485527 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.494077 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.526160 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.527755 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.533324 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:43.539110 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:43.542181 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.547569 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.548097 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.556361 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.562273 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.562470 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.578188 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.592198 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.596072 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.597164 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.598860 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.606043 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.612872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.614861 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.625609 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:43.691610 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.722967 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:43.728440 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.729305 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.735734 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.753829 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.754644 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.757995 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.758564 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:43.773511 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:43.805012 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.809350 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.810385 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:43.821087 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:43.824769 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.836640 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.847554 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.852182 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.863310 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:43.863672 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:43.882212 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.882290 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:43.905330 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:43.915449 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:43.927559 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:43.981142 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:01:43.985770 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:43.988543 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:43.993559 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:44.002404 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.002586 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.009258 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.009758 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.010047 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.099424 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.101984 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.102248 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.111840 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.113050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.138757 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.139341 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.150453 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.151355 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.160634 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.162866 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.164359 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.164896 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.201922 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:44.214277 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.242738 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.246832 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.253751 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.268018 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.281162 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.284808 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.293936 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.323416 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.325654 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.342532 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.343456 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.346050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.349216 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.366462 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:44.373803 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:44.374115 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.378065 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.378652 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.391975 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.420776 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.420776 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.429420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.431123 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.455720 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:44.473505 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.488402 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.488731 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.489318 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.497941 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.501919 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.511857 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.514585 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.540719 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:44.555963 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.580793 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.582762 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.582986 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:44.591331 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.593294 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.602026 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.607218 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.608428 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:44.634721 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.714433 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.725916 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.726197 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.735215 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.756254 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.763002 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.764642 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.770177 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.773195 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.784080 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.823009 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.827548 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.840744 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.849364 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.851480 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.859056 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.861470 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.862762 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.863889 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.870370 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.880642 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:44.888262 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.894102 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:44.896355 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.901356 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:44.917551 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:44.932148 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:44.935513 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:44.940665 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:44.965687 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:44.976004 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:45.015978 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.018559 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.019232 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.020851 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.023147 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:45.032400 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 08:01:45.045536 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.049691 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:45.056124 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:45.083699 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:45.137060 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.137058 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.159111 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.159994 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.169342 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.172811 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:45.176303 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:45.178264 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.238332 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.242888 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.244063 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.245697 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.259927 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.287298 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.290659 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.370081 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.376282 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.378578 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.393183 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.400048 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.400736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.410027 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:45.469875 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:45.470978 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.471158 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.477217 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.492224 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.504024 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.522503 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.524050 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.525657 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.530107 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.532178 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.535251 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.541956 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.561197 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:45.564553 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:45.565205 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.602767 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.605834 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.618252 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.618558 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.619237 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.619834 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.625818 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.635201 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.635433 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.649958 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.665309 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.670892 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.683643 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.692531 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.762570 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.770530 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.770617 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.770967 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.771018 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.771325 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:45.771950 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:45.810910 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:45.836678 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:45.858810 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.859243 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.867982 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.877398 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.878543 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:45.882315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:45.890898 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.911150 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.918104 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:45.928845 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:45.935649 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.935824 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:45.939008 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:45.948585 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:45.997117 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:46.077472 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.078454 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.090641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:46.094094 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:46.102106 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.116122 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.119705 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.125496 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.137492 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.137889 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.139519 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.155164 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.171391 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:46.173341 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:46.185861 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.195472 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.199615 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.209419 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.210155 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.224881 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.243096 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.245750 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.249457 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:46.255695 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:46.293499 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:46.300686 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.307080 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.310731 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.313179 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.314833 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.320116 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.328255 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.402495 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.407356 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.420938 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.440507 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.449118 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.449849 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.452260 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.459110 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:46.464221 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:46.466115 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.466361 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.467293 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.468093 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.488029 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:46.512421 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:46.517616 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.568954 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.569762 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.576555 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.579483 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.584110 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.602644 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.605443 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.653048 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.660869 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.668538 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.683622 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.686350 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.690562 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.693377 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.701020 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:46.714201 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:46.717914 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.717929 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.720894 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:46.732350 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:46.744740 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.759665 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.791222 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.858143 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.859345 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.869442 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.874114 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.875247 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.876356 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:46.884701 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:46.884950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:46.892349 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.895915 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:46.903532 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:46.921178 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:46.943189 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:46.943521 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:46.951983 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.013839 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:47.017902 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.020119 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.024713 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.025640 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.026871 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.030185 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.051662 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:47.071965 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:47.120549 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.134209 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:47.142393 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.143651 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.151752 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.157785 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.168087 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.174044 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.183055 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.201756 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.201783 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:47.219341 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.221438 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.230754 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.232152 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.233616 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.234088 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:47.264439 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:47.271330 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.361954 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.366732 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.370607 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.392165 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.392899 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.398818 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.400895 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.412279 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.415585 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.417673 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.419812 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.428494 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.431094 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.433241 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.434623 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.441313 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.445688 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.457681 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.465832 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.484259 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.494063 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:47.539181 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:47.608597 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:47.624960 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:01:47.671828 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.683988 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.685354 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.696141 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.697430 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.704267 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.708513 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.718780 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.731442 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.733736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.751102 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.751361 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.766247 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.766523 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.781715 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.781881 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.800952 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.801115 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:47.804421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:47.807345 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:47.812228 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.819087 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.829247 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.831585 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.842610 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.844480 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.862102 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.864784 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.869067 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:47.878126 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.882367 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:47.888675 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 08:01:47.891863 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:47.891885 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:47.905827 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:47.918263 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:47.948064 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.051918 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.053689 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.060837 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.065027 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.070614 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.072768 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.076813 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.097119 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.113003 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.114555 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.116144 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.124132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.128842 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:48.151286 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.158666 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:48.226534 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:48.263625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.272166 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.279465 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:48.288693 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:48.292433 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:48.295466 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.297839 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.307965 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.308187 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.343138 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.345675 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.352253 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.369518 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.369753 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.379738 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.387113 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.389873 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.393349 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.399531 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.406059 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.418493 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.423177 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.453133 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.501920 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.506785 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.509260 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.510051 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.519335 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:48.524132 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:01:48.529242 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.530090 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.531440 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.533986 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:48.559216 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.559306 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.560570 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.566370 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.579080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.586493 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:48.665140 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:48.669313 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.671694 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.685461 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.685823 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.690769 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.696511 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.700016 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:48.720106 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:48.744336 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.755559 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.759001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.759701 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.759881 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.777128 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.792234 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.822762 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.829602 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.834216 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.845844 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.847306 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.847313 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:48.847450 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.848452 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.910707 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.912801 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:48.913503 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:48.914862 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:48.923365 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:48.926074 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:48.927984 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:48.928766 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.999059 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.001075 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.012436 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.013444 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.015105 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.022583 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.026119 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.112730 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.117219 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.117992 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.123081 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.124930 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.131737 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.138088 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.148947 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.150627 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:49.163081 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:49.167423 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.169693 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.171347 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.174506 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:49.189607 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:49.191190 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:49.203233 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:49.251667 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.268111 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.276094 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.282438 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.283776 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.292613 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.311388 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.340374 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:49.348249 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:49.380823 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:01:49.391078 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.395740 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.406477 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.411142 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.426239 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.426719 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.438640 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:49.477876 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.496873 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.498281 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.499578 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.504127 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.505455 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.528178 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.535792 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.540149 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.551954 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.556741 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.557008 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.566366 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.579586 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.583734 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:49.606109 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.620322 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.627367 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.630512 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.630973 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.639108 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.645812 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.664806 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:49.692705 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.710680 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:49.712606 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.713142 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.718024 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:49.731727 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:49.734251 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.737095 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.746033 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:49.749624 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:49.750805 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:49.783582 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:49.806828 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.809612 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:49.811700 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.815438 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.818562 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.824182 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.833158 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.922328 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.936900 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.947378 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:49.956487 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:49.987725 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:49.989502 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:49.989869 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:49.989871 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.009382 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.013125 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.032184 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.051573 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.065042 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.072297 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:50.085211 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:50.087314 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.087339 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.087874 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.102578 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.131971 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.132930 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.141119 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.142375 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.145115 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.156708 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.159328 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.163703 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.173311 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.192186 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:50.233660 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:50.238996 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.238998 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.243041 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.244534 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:50.248198 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:50.274309 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.285988 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.296484 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.370093 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.374480 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.377952 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.390221 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.393020 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.403267 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.406250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:50.416534 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:50.423157 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:50.471346 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.472903 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.484341 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.493623 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.497171 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.500607 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.509863 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.558347 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.562829 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.567008 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:50.568815 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:50.569545 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.574064 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.577500 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.584837 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.589130 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.589988 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.600023 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.617630 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.622020 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:50.631561 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:50.649330 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.650707 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:50.679548 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:50.711298 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.713900 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.723224 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.726254 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.737196 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:50.741945 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:50.755009 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.763598 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.785701 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.799041 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.801390 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.804970 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.809739 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:50.824044 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.845016 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.855581 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:50.857605 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.857858 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:50.868158 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:50.869658 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:50.871783 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:50.892638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:50.948549 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.000074 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.013621 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.014343 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.035315 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.038619 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.055449 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.061787 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.097690 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.106982 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.119267 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.128278 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.132285 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.133316 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:51.134043 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:51.138749 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:51.147840 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:51.157383 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.159567 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.166172 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.183260 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.187455 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.192958 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.194437 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:51.196316 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.196388 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.197108 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:51.202633 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.211366 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.211661 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.222765 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.237196 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.243942 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:51.257347 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.317658 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:51.322973 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:01:51.338450 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.343895 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.352999 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.370870 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.371155 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.382108 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.430876 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.437597 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.444494 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.451715 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.453935 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.476588 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.479632 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.487771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.488200 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.490906 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.512815 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.524788 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:51.535869 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:51.543651 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.560227 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.560464 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.570660 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.570909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.571996 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.575268 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.577632 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.581674 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:51.617033 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:51.690296 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.702015 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.711699 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.713002 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.725167 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:51.731315 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.735698 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.742906 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.746168 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:51.749522 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.755055 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.756922 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.760360 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.763037 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:51.770661 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.774312 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.775480 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.775727 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.777807 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.787400 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.791121 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:51.794878 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.797250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.831552 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.856462 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:51.931063 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:51.951877 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:51.959151 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:51.960121 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:51.966879 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:51.967003 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:51.986794 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.006171 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.024302 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.029924 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.029933 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.037684 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.042909 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.061726 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:52.062969 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:52.067840 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:52.107203 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:52.139639 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:52.164465 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:52.193795 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.196899 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.197404 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.216646 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.231108 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.232239 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.232287 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.264918 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.265599 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.266654 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.278719 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.297200 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.307500 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.318493 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.319618 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.324156 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.339051 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.339211 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.350391 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.351536 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.360630 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.366386 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.401065 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.407505 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.413823 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.418164 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.418869 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.427958 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.428082 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.451097 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.451475 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:52.452893 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:52.453954 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.455550 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.457909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.466645 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.579079 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.584144 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.584537 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.590231 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.606168 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.606830 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.632380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.632860 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.635329 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.644932 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.655148 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.662999 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.664891 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:52.671006 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:01:52.678289 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:52.685173 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:52.734900 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:52.784088 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.795584 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:52.795727 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 08:01:52.805451 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.805438 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.814045 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.858412 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.861712 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.867516 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.882520 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.883953 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.886189 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.899789 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.905716 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.921865 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:52.922250 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:52.929517 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:52.933933 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:52.971524 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:52.981956 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:52.982908 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.004916 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.048557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.056535 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.061159 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.063484 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.065476 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.079335 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.086268 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.093351 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.096170 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.096518 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:53.103932 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:53.114123 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.120951 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:53.135871 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:53.145409 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.155191 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.253746 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.257658 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.283563 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.283949 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.286592 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.303372 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.323685 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:53.330457 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:53.334578 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.334837 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.338402 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.361817 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.362850 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.370999 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.376667 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.387501 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.401597 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.403203 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.403582 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.414460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.438095 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.451753 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.454298 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.460036 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.467120 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.475529 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.475946 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.484495 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.504036 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:53.519435 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:53.577170 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.583106 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.589151 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.602052 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.603327 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:53.607434 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:01:53.610137 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:53.612614 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.616562 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.616845 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.618849 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.620601 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:53.626996 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:53.641021 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.654286 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.658171 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.665601 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:53.711138 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.716013 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:53.716909 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.723654 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.724970 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.726775 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.734095 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.745859 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.748037 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.858430 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.860925 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.863981 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.872884 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.883402 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.886003 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.896569 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.916051 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:53.922089 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:53.922721 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:53.922798 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:53.925621 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:53.935655 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:53.940891 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:54.000972 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.020383 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:54.024741 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.029356 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.029377 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:54.048903 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:54.049703 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.054767 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:54.060139 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:54.065093 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.079980 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.132262 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.132819 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.136449 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:54.143702 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:54.154031 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.154099 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.164857 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.191465 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.191573 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.206439 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.207788 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.219822 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.231925 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.239948 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.248556 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:54.273650 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.297634 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:01:54.299496 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:54.311607 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.322804 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.327576 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.343892 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.344264 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:54.347182 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:54.347465 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.380509 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.384114 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.388713 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.400161 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.403026 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.403754 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.425278 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.430441 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:54.437659 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:54.471098 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.471589 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:54.473554 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.483275 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.484264 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.486029 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.488842 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.502861 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:54.510994 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:54.538530 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.545282 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.558516 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:54.564113 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:54.564559 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.567816 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.570554 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.576301 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.588584 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.645341 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.646804 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.653764 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.660003 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.663669 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.680603 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.681949 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.743391 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.746117 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.757829 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.766252 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.767794 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.787894 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.788759 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.789507 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.810046 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.811381 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.818153 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:54.818501 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:54.853818 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.866250 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.878556 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:54.882268 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:54.884838 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:54.886301 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.893555 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:54.919097 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:54.931677 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:54.934862 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:55.019654 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:55.046485 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.058779 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.067523 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.077908 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.081346 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.087599 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.100790 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.102586 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.116552 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.124787 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.134130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.142455 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:55.148653 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:55.155452 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.157589 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.160066 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.166265 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.176890 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.181735 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.205379 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.207782 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.215680 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.298672 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.299129 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.301031 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.303329 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.315610 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.318459 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.329582 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.330598 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:55.344168 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 08:01:55.350250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.354872 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.368055 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.397110 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.425668 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.427418 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:55.428748 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:01:55.441596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.449786 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.455119 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.456104 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.474946 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.477032 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:55.485595 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:55.499727 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:55.519114 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:55.569694 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:55.595007 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.596311 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.598717 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.620785 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.630890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.639045 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.639078 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.671938 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.681236 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.689450 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.696768 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.703088 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.710903 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.732967 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.745162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.767802 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.783443 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.784835 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.786651 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:55.788516 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:55.798341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:55.799933 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:55.817433 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.818747 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.820531 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.826880 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.836156 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.845294 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.846578 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.859035 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:55.895700 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:55.912336 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.916067 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.925623 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.927124 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:55.929961 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:55.933612 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:55.934020 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.936439 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:55.960750 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:55.970564 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:55.991664 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:56.016845 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:56.025791 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.026342 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.029776 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.106351 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.109859 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.115098 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.130179 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.139926 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.141438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:56.146731 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:56.147712 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.155274 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.155691 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:56.159921 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:56.160346 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.166085 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.167585 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.170936 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.196671 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.301952 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.306272 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.327304 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.334509 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.339350 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.339552 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.343537 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.357023 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.366279 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.368206 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.370206 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.370344 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.372265 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.381687 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.463618 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.464483 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.465380 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.468158 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.470007 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.470636 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.495117 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:56.505270 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.506043 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:56.507264 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.507732 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.509261 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.510671 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.511504 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:56.520336 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.528126 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:56.548014 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.554957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:56.619096 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.641396 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:56.647063 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.651144 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.665979 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.667397 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:56.680608 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:56.683951 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.684504 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.693679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.704562 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.709228 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.713671 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.733649 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.736665 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.743425 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.744318 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.776409 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.792809 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.793378 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.804898 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.806203 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.808621 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.809666 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.880816 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:56.884039 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:56.887470 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.899622 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:56.915334 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:56.917194 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:56.918205 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.924006 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:56.926822 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.931351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:01:56.941071 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:56.942096 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.967077 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:56.986248 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:56.992806 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:56.993945 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.994277 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:56.994527 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:56.997728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.090976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.099263 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.102262 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.110342 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.129050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.133265 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.135078 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.225471 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.225471 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.238364 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.242656 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.246359 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.247540 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.249792 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.255534 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.256404 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.256899 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:57.265481 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:57.301034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:57.309609 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:57.323942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.324090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.324679 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.335495 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.349490 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.355795 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.359200 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.359524 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.403894 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.415867 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.482313 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.486110 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.488597 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.494627 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.503314 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.522140 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.522545 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.539350 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.541304 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.553031 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.555310 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.565260 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.566375 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.573096 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.576107 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.583771 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.587908 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.589314 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.604584 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.606885 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.612195 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:57.624748 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:57.719655 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.720374 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.722715 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.722870 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.723937 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.725195 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:57.731047 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 08:01:57.737829 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:57.786074 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 08:01:57.819806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.824345 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.832616 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.841110 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.842844 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.853217 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.855036 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:57.882551 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:57.885569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.885977 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:57.892377 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.892805 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.896989 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.898231 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:57.908893 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.915088 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:57.915206 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:57.916769 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:57.922909 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:57.929886 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.945492 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:57.953066 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:57.956964 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.099550 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.101151 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.117594 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.119897 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.123375 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.140551 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.142164 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.145448 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.146890 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.155220 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.165256 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.176635 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.176992 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.199916 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.236599 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.253156 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.254251 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.268477 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.275081 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.284991 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:58.289898 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:58.292605 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:58.317377 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:58.344657 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.350824 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.351550 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.352535 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.360294 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.363726 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.365633 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.368448 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.369520 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.392803 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.419687 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.420537 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.421691 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.429910 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:58.448602 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.461391 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:01:58.466593 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.468901 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:58.524596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.525514 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.526789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.536847 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.540794 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.541240 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.542065 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.672710 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.686473 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.687522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.696168 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.705682 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.711745 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.715107 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.716645 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.717065 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.717507 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.744499 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:58.757762 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:01:58.767287 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:58.768918 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.779034 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.783118 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.790006 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.793220 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.808022 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.811554 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.818534 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:58.848840 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:58.855176 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:58.859699 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.901243 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.906154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:58.911051 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:58.912422 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:58.914372 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:58.933120 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:58.933852 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:58.953274 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.960147 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.967732 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:58.997568 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.006793 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.015033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.020175 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.023510 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:59.035581 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:59.048661 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.050817 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:59.142220 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:59.165871 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.173846 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.178819 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.202257 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.206182 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:59.217994 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:59.230064 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.249132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.257707 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.262516 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.265267 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:01:59.282641 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:59.292215 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:59.305719 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:59.309075 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.309217 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.315640 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.317873 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.333845 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.335639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.336705 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.346071 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.350188 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:01:59.361963 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:59.371088 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.434286 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.436371 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.444643 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.449476 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.454429 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.460863 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:59.462671 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:01:59.465477 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.520946 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.524802 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.526062 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.534708 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:59.547540 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:01:59.559262 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.561987 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.563385 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.565205 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.567092 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.569053 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.569085 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.571239 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.589459 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.594157 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.700070 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.703107 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.707464 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:01:59.726593 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:59.732935 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.740530 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.742029 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.759864 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.761289 12662 maintenance_manager.cc:419] P dc2c479861bf4184ba4d8446b0c8d4cf: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033403
I20240503 08:01:59.764912 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.770618 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.775216 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.782256 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.782456 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:01:59.786782 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.787699 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:01:59.800966 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:01:59.806444 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:01:59.807654 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:01:59.809000 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.822883 12661 maintenance_manager.cc:643] P dc2c479861bf4184ba4d8446b0c8d4cf: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.022s sys 0.000s Metrics: {"bytes_written":20467,"cfile_init":1,"dirs.queue_time_us":9535,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15405,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"rows_written":43,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":5469}
I20240503 08:01:59.827414 12662 maintenance_manager.cc:419] P dc2c479861bf4184ba4d8446b0c8d4cf: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7539 bytes on disk
I20240503 08:01:59.828023 12661 maintenance_manager.cc:643] P dc2c479861bf4184ba4d8446b0c8d4cf: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240503 08:01:59.835729 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.845293 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.849601 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:01:59.852845 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.874953 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.979825 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:01:59.980233 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:01:59.982548 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:01:59.996091 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:01:59.999751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.031807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.039924 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.053519 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.058836 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.062062 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.085548 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.085633 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.095935 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.096947 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.128800 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.134989 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.141134 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.142562 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.146624 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.150998 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.158629 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.160811 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:00.161624 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:00.166724 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.185451 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.191740 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.192340 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.192593 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:00.195384 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:02:00.196074 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:00.222805 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:00.239163 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:02:00.395656 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.396711 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.397236 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.397428 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.408152 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.410372 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.410827 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.413810 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.431057 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.436066 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.446661 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.468775 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.473716 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.474201 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.480343 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.489312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.495918 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.501066 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.512055 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.515450 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.517294 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.532867 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.535519 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.536463 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.547217 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.548887 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:00.571318 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:00.573827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.586777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.591878 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:00.593322 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.608259 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:00.614598 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.618425 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.629329 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.637924 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.644964 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.660079 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:00.725648 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:00.736725 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.737210 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.740476 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.740711 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.744185 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.748525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.761150 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:00.766063 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.788969 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:00.850239 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:00.857954 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.861146 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.865218 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.869792 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.873260 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:00.883219 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.891341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:00.972579 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:00.978432 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:00.978734 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:00.982439 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:00.990161 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:00.991973 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:00.993755 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.042155 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.045800 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.060530 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.063930 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.075942 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.085916 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.092058 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.118387 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.152563 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:01.168408 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:01.168812 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.177898 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.183610 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.194129 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.203059 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.203510 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.212556 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.224114 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.231362 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:01.237249 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:01.241405 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.245501 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.258216 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.271950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.315225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.325289 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.329192 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.338943 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.344275 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.356448 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.359589 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:01.394551 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.434257 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.434438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.436884 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.440759 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:01.443925 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:01.444221 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.447877 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.466908 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.478113 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.494375 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:01.500124 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.504098 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.504261 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.508174 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.518751 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.521996 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.545401 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.578351 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.580595 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.581269 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.585052 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.596280 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.604643 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.635929 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.649886 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.653888 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.656368 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.664820 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.669411 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.672148 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.712991 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.745407 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.748726 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.758884 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.762513 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.763605 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:01.768404 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:01.769963 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.778117 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:01.799780 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.808265 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.814889 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.815797 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:01.816341 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.817564 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.818843 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.827940 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.833148 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:01.853402 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:01.902899 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:01.924014 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:01.924291 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:01.924564 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:01.948032 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:01.957904 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:01.975467 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:01.996150 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.008030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.019934 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.031396 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.035739 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.038192 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.041482 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.061442 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:02.082423 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:02.123914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.133065 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.137530 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.139111 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.148761 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.169139 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.192425 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:02.208606 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:02.237623 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.244292 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.244524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.256470 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.263926 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.275866 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.278139 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.280285 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.291975 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.296520 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.305325 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.308238 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:02.318737 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:02.321740 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.326412 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.335955 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.343770 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.348981 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.350144 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.365988 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.373885 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.402983 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.406216 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.417663 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.417999 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.428985 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.442214 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.450059 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.463793 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:02.471086 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:02.543426 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.547102 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.549131 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.559934 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.560106 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.582273 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.583961 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.607627 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.613488 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.613710 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.623817 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.628935 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.638950 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.650302 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:02.686478 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:02.753235 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.766114 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.769054 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.775768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.779476 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.780947 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.790998 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.791513 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.802783 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.804307 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.807313 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:02.812002 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:02.839118 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:02.846496 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:02:02.849426 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.853116 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.903734 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.909773 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.910264 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:02.921614 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.925307 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:02.932726 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.934561 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:02.938608 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:02.953292 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 08:02:02.963950 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.972090 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:02.977448 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:02.987334 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:02.991153 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:02.993124 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:02.994315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:03.061811 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.062966 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:03.095511 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.102475 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.102691 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.104075 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.125375 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.126854 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.143126 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.183876 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.188030 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.189235 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.202093 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.207549 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:03.216055 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:03.231276 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.233387 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.235713 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.240104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.243067 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.263056 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.269207 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.269430 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.277139 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:03.377763 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:03.390087 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.409802 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.417486 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.417655 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.428167 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.441633 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.447088 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.455921 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.457224 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.465996 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.469024 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.472064 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.473766 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.479642 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.494259 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.503139 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.505564 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.538518 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.548007 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.552277 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.567580 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.644076 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.644578 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.647059 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:03.652405 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:03.660135 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.662335 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.681771 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.684876 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.689749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:03.693434 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:03.697085 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:03.706604 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:03.710566 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.710958 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.711287 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.721678 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.739094 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.772302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.788440 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:03.793205 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.798617 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:03.801349 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.815508 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.817363 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.915247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:03.925380 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.928920 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:03.933691 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:03.936733 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:03.954066 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:03.960333 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:02:03.961335 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:03.972647 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.986951 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.012467 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.018393 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.021870 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.026681 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.038393 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.039373 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.040035 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.048910 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.054386 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.059659 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:04.064127 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.066071 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.076714 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.077613 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.097062 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.102885 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.105388 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.112818 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.122138 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.126188 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.145535 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.155517 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.171746 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.172850 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:04.196190 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.198486 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.207890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.208636 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.213791 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.223476 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:04.230278 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:04.233810 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.238477 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:04.344064 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:04.350978 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.355161 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.367664 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.380790 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.385226 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.387084 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.396638 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.398716 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.400884 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.416708 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.434222 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.438117 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.457345 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.458500 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:04.497107 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.559441 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.563189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.568220 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:04.578765 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:04.584282 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.585517 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:04.602275 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:04.605645 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.617441 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.622031 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.624311 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.634320 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.640188 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:04.665833 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:04.674156 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.677661 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.678833 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.682809 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.686442 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.689857 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.700795 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.707871 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.721127 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.731104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.741524 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.750627 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.757166 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.760897 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.764670 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.784636 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.799218 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1): perf score=0.033337
I20240503 08:02:04.836035 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.838291 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.038s user 0.029s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":14040,"dirs.queue_time_us":4444,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1939,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":856,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":213,"threads_started":3,"wal-append.queue_time_us":129}
I20240503 08:02:04.839713 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1): 4923 bytes on disk
I20240503 08:02:04.840701 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:02:04.843254 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(9e7f229898c24572815177628720521f): perf score=0.033348
I20240503 08:02:04.869060 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(9e7f229898c24572815177628720521f) complete. Timing: real 0.026s user 0.023s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1840,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":207}
I20240503 08:02:04.870301 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f): 4581 bytes on disk
I20240503 08:02:04.870796 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240503 08:02:04.910636 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:04.916641 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:04.919891 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa): perf score=0.033372
I20240503 08:02:04.922346 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:04.942482 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.945588 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:04.954042 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.974826 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1): perf score=0.033387
I20240503 08:02:04.979669 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:04.980345 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:04.980845 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:04.987026 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:04.998163 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.036295 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.042349 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.067173 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.076005 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.078368 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:05.114687 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.194s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":8428,"dirs.queue_time_us":10713,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":3200,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":321,"threads_started":3,"wal-append.queue_time_us":186}
I20240503 08:02:05.115975 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa): 4838 bytes on disk
I20240503 08:02:05.116737 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.123113 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.148s user 0.030s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":10838,"dirs.queue_time_us":13499,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1318,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":288,"threads_started":3,"wal-append.queue_time_us":13814}
I20240503 08:02:05.127539 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1): 4923 bytes on disk
I20240503 08:02:05.128638 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 08:02:05.132282 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:05.159940 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(d8994c38d0f640e9bd5b5c322c603b9f): perf score=0.033337
W20240503 08:02:05.215739 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:05.303336 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.307684 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:05.322021 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:05.325013 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.349419 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:05.363489 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:05.368137 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(a924487225dc4246aeb695060084096b): perf score=0.033393
I20240503 08:02:05.373947 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57): perf score=0.033392
I20240503 08:02:05.380815 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(a924487225dc4246aeb695060084096b): perf score=0.033399
I20240503 08:02:05.387396 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:05.400885 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:05.427115 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:05.436241 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:05.442219 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 08:02:05.450453 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.466305 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.092s user 0.020s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5600,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":2013,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":207,"threads_started":2,"wal-append.queue_time_us":10461}
I20240503 08:02:05.467326 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57): 4583 bytes on disk
I20240503 08:02:05.468386 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.100s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":10112}
I20240503 08:02:05.468945 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.469221 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b): 4923 bytes on disk
I20240503 08:02:05.469842 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.471189 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.090s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":27596,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1259,"drs_written":1,"lbm_read_time_us":1229,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":12446}
I20240503 08:02:05.470835 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.471737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:05.472196 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57): perf score=0.033423
I20240503 08:02:05.491685 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:05.517020 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:05.536939 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:05.546092 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(d8994c38d0f640e9bd5b5c322c603b9f) complete. Timing: real 0.383s user 0.033s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":26241,"dirs.queue_time_us":97642,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":5986,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":66363648,"thread_start_us":286,"threads_started":3,"wal-append.queue_time_us":14650}
I20240503 08:02:05.547719 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f): 4838 bytes on disk
I20240503 08:02:05.548436 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.565539 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.089s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":42537,"compiler_manager_pool.run_cpu_time_us":40,"compiler_manager_pool.run_wall_time_us":39,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":46313}
I20240503 08:02:05.566535 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b): 4923 bytes on disk
I20240503 08:02:05.567000 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.569335 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57): 4583 bytes on disk
I20240503 08:02:05.569743 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.573957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:05.578420 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:05.606300 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:05.620765 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.625006 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:05.626084 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:05.629386 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.639420 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.676790 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:05.681196 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.693106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:05.703900 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:05.717826 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.723894 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.742972 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.743396 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:05.749029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:05.754261 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.761241 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.761504 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:05.772384 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:05.791961 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:05.802654 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.810158 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.821183 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47): perf score=0.033346
I20240503 08:02:05.827502 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.869901 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.049s user 0.013s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1332,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":3317}
I20240503 08:02:05.870788 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47): 4838 bytes on disk
I20240503 08:02:05.871192 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:02:05.939479 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:05.960247 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:05.965418 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:05.966104 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:05.971577 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(70c5de226c7a4b349eef86a79d172851): perf score=0.033392
W20240503 08:02:05.974390 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:05.976796 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:05.978456 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): perf score=0.033394
I20240503 08:02:06.000487 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:06.018592 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:06.021847 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.024207 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.024497 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:06.038606 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.046648 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.053822 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.075s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1284,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":7353}
I20240503 08:02:06.054813 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47): perf score=0.033414
I20240503 08:02:06.056586 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.061666 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.065032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.071857 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.081516 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.091825 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.092494 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.104559 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.041s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":200,"threads_started":2,"wal-append.queue_time_us":12518}
I20240503 08:02:06.105383 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): 4923 bytes on disk
I20240503 08:02:06.105752 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.106348 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47): 4838 bytes on disk
I20240503 08:02:06.106659 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.130842 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.135238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.142261 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.170s user 0.021s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13501,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":6175,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2334,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":114920}
I20240503 08:02:06.143747 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851): 4583 bytes on disk
I20240503 08:02:06.144485 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.150769 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(b094d04434d24923806b0e9e209c3fca): perf score=0.033337
I20240503 08:02:06.159158 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.169284 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.199965 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.049s user 0.012s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14771,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1750,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":10822}
I20240503 08:02:06.200799 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca): 4923 bytes on disk
I20240503 08:02:06.201233 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.210691 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.219897 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.229113 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.231487 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.236433 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.243795 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.274863 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.301499 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.301733 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(b094d04434d24923806b0e9e209c3fca): perf score=0.033381
W20240503 08:02:06.307731 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:06.320266 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.320516 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.320958 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.324730 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.342505 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.343605 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.346539 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:06.352571 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.356279 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.054s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39266,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":36276}
I20240503 08:02:06.357894 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(02d3e4066cd14aa18619310a4490246b): perf score=0.033393
I20240503 08:02:06.360621 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca): 4923 bytes on disk
I20240503 08:02:06.361076 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.361922 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(c917c5849f54411eb55d69ab33456bf4): perf score=0.033397
I20240503 08:02:06.383836 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(c917c5849f54411eb55d69ab33456bf4) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1145,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 08:02:06.384769 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4): 4838 bytes on disk
I20240503 08:02:06.385162 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.393008 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.035s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15829,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1172,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":895}
I20240503 08:02:06.393874 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b): 4583 bytes on disk
I20240503 08:02:06.394325 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240503 08:02:06.425726 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.428413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.429855 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.430495 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.434517 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:06.437785 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:06.445361 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.448884 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.476609 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.507899 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.517980 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.519753 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:06.522953 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.524277 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.535439 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.546715 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.552596 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:06.570816 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:06.584455 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.650676 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.651942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.659554 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.660300 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:06.661335 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:06.671105 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.671667 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.676977 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.703464 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d): perf score=0.033348
I20240503 08:02:06.777382 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.781011 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.782254 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.784371 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.786399 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.795459 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.800680 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.801205 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.802779 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.805509 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.810329 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.812886 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:06.821379 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:06.823433 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.828605 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.829826 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:06.837730 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.837730 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.842166 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.868978 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.870230 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.876857 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45): perf score=0.033394
I20240503 08:02:06.881552 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.178s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16795,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":7764,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19611,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":6190}
I20240503 08:02:06.882423 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d): 4922 bytes on disk
I20240503 08:02:06.883134 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:02:06.887964 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(48e4f2a993164fb6b1df674311698006): perf score=0.033395
I20240503 08:02:06.891717 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:06.903615 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.914757 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:06.931303 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.935865 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.940699 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:06.977222 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.011797 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.135s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":72919,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":2149,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":628,"threads_started":2,"wal-append.queue_time_us":70811}
I20240503 08:02:07.012562 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45): 4838 bytes on disk
I20240503 08:02:07.012900 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 08:02:07.028262 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:07.029278 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.141s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12142,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":6136,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":112,"threads_started":2,"wal-append.queue_time_us":1831}
I20240503 08:02:07.030254 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006): 4584 bytes on disk
I20240503 08:02:07.030795 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:02:07.043781 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:07.079473 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:07.121635 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:07.129452 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:07.134521 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(076781af2ac14ddc87d31e3ec77db08f): perf score=0.033350
I20240503 08:02:07.135499 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.142030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.147557 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(f1151731adc9407b86d143d6d41f69b4): perf score=0.033354
I20240503 08:02:07.163074 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.168731 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.168751 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.178210 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.190845 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.202809 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.205247 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.232033 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.236469 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.237712 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.246170 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.265085 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.117s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2920,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":35011,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":38317}
I20240503 08:02:07.266155 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4): 4838 bytes on disk
I20240503 08:02:07.266136 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(d44b2c67014742af9a65a3576f509702): perf score=0.033387
I20240503 08:02:07.270584 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.283776 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(d44b2c67014742af9a65a3576f509702): perf score=0.033395
I20240503 08:02:07.301613 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(076781af2ac14ddc87d31e3ec77db08f) complete. Timing: real 0.167s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1979,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":2202,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":7174}
I20240503 08:02:07.302753 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(f1151731adc9407b86d143d6d41f69b4): perf score=0.033395
I20240503 08:02:07.305311 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.021s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1229,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":156}
I20240503 08:02:07.307718 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702): 4923 bytes on disk
I20240503 08:02:07.311473 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.342530 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.344702 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.362802 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.370703 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.372583 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.387362 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.411775 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.428719 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.429145 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.163s user 0.017s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3055,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1211,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1378,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":110421}
I20240503 08:02:07.430814 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702): 4923 bytes on disk
I20240503 08:02:07.431439 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.440649 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.441066 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.441129 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.448175 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.448976 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.470474 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.472813 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.494325 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.502074 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.510421 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.511889 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.519636 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.217s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":16110}
I20240503 08:02:07.520706 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4): 4838 bytes on disk
I20240503 08:02:07.525008 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.525836 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f): 4582 bytes on disk
I20240503 08:02:07.526223 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.540764 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.541509 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:07.558002 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:07.582249 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:07.614344 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.616156 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:07.616598 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:07.625288 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.633404 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.641638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:07.641806 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:07.644760 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.654686 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.682811 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.689426 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.697285 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.707427 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.710705 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.718680 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.724944 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.774194 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(047aba81fad543ce8b562a184746b163): perf score=0.033348
I20240503 08:02:07.777715 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3): perf score=0.033346
I20240503 08:02:07.797303 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":574,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1335,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":1295}
I20240503 08:02:07.798218 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3): 4583 bytes on disk
I20240503 08:02:07.798825 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.802075 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(047aba81fad543ce8b562a184746b163) complete. Timing: real 0.028s user 0.013s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11722,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1060,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":297}
I20240503 08:02:07.803409 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163): 4923 bytes on disk
I20240503 08:02:07.803894 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.808390 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.814088 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(047aba81fad543ce8b562a184746b163): perf score=0.033356
I20240503 08:02:07.820178 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.822585 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.829977 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:07.836248 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:07.837378 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.838234 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:07.850256 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:02:07.876567 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(047aba81fad543ce8b562a184746b163) complete. Timing: real 0.062s user 0.011s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15204,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1090,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":11188}
I20240503 08:02:07.877580 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163): 4923 bytes on disk
I20240503 08:02:07.877995 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.878793 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190): perf score=0.033373
I20240503 08:02:07.905220 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.026s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1373,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":133}
I20240503 08:02:07.906167 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190): 4838 bytes on disk
I20240503 08:02:07.906530 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240503 08:02:07.933589 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:07.935097 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:07.937644 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:07.937917 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:07.944522 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:07.985103 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.993785 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.001214 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.006079 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.027944 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.048367 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.057842 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.058216 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.058591 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.061203 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:08.068715 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:08.083655 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:02:08.089812 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:08.093116 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:08.109462 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.113874 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.119799 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.124470 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.125083 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.129288 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:08.140424 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.150851 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.200230 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.200887 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.204530 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.205341 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.209194 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.222573 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.226382 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:08.246562 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.301384 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(a45748a98c3447d3aa2e8de94981a745): perf score=0.033367
I20240503 08:02:08.307227 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6): perf score=0.033373
I20240503 08:02:08.309598 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.319496 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.323869 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.342983 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.346299 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.363773 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.373266 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.394405 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.409564 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(a45748a98c3447d3aa2e8de94981a745): perf score=0.033398
I20240503 08:02:08.431269 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.432360 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.131s user 0.014s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2639,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":13965,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":211,"threads_started":2,"wal-append.queue_time_us":6468}
I20240503 08:02:08.433435 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745): 4838 bytes on disk
I20240503 08:02:08.434015 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:02:08.436585 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.439321 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.445478 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.448516 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.459980 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.470885 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.474247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.479116 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.482168 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.484040 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.496567 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.509451 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.202s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":22041,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":99316}
I20240503 08:02:08.510406 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6): 4923 bytes on disk
I20240503 08:02:08.510876 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:02:08.515261 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.519299 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.110s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":92363,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":90184}
I20240503 08:02:08.520255 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745): 4838 bytes on disk
I20240503 08:02:08.520725 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240503 08:02:08.525945 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:08.530392 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:08.616710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.644310 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:08.645350 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.648305 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.655375 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.659879 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.665652 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.675335 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.686165 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.687032 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.687240 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(5212dff7ded143a7923bb6176466b0fa): perf score=0.033352
I20240503 08:02:08.690747 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.691640 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(7d61aa445a50417b82916019c2a4298c): perf score=0.033352
I20240503 08:02:08.701282 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.706023 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.721065 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":413,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":6238,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":825,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":149}
I20240503 08:02:08.722044 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c): 4923 bytes on disk
I20240503 08:02:08.722491 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:08.741473 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.748301 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.061s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1895,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":2855,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":19221}
I20240503 08:02:08.749224 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa): 4583 bytes on disk
I20240503 08:02:08.749694 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:02:08.758419 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.762348 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(5480e544dca3428ab4fcf190577d5331): perf score=0.033373
I20240503 08:02:08.768484 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.772377 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(7d61aa445a50417b82916019c2a4298c): perf score=0.033376
I20240503 08:02:08.784574 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.785373 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.785936 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.803490 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:08.819116 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.820389 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.821064 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.844760 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.852105 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.090s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":17607}
I20240503 08:02:08.853595 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331): 4836 bytes on disk
I20240503 08:02:08.854262 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:08.858865 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:08.864673 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.866031 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:02:08.869604 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.097s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":860,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1829,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":22501}
I20240503 08:02:08.870643 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c): 4923 bytes on disk
I20240503 08:02:08.874316 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:08.880901 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:08.881250 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:08.901011 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 08:02:08.918254 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.935354 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:08.987376 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:08.988987 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.006644 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.010627 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.014154 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:09.035776 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:09.038074 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:09.072048 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:02:09.073424 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:09.073724 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.080404 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:09.092845 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:09.095772 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.106267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.121441 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.123188 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.127027 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.127418 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.131003 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.136869 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.139130 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.153792 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.158027 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.168780 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.170617 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.175182 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.182011 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.185457 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.189937 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.193375 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.217839 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:09.226017 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:09.238467 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.251701 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1): perf score=0.033335
I20240503 08:02:09.298599 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.047s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1446,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":16354}
I20240503 08:02:09.299459 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba): perf score=0.033347
I20240503 08:02:09.326028 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.026s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":3280,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":2304,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":6206}
I20240503 08:02:09.326805 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba): 4923 bytes on disk
I20240503 08:02:09.327268 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 08:02:09.327910 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1): 4583 bytes on disk
I20240503 08:02:09.328203 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:09.361483 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.364341 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.368512 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e): perf score=0.033367
I20240503 08:02:09.370492 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.385728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.387104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.388484 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.391213 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.023s user 0.012s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":197,"threads_started":2,"wal-append.queue_time_us":179}
I20240503 08:02:09.395601 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e): 4838 bytes on disk
I20240503 08:02:09.396654 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:02:09.396975 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.462149 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.470115 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.470376 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.474038 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.475718 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba): perf score=0.033394
W20240503 08:02:09.483072 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:09.491847 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.495883 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.499177 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.501159 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.502784 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.508797 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.543777 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.549799 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.550894 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.563052 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:09.567462 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:09.578716 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.103s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4652,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1813,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":5108}
I20240503 08:02:09.580494 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba): 4923 bytes on disk
I20240503 08:02:09.591295 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.595377 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 08:02:09.610191 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.614019 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.617611 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.617944 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.623984 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.650653 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:09.745812 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:02:09.755373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.756984 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.763422 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.770800 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.772836 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.773634 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:09.795485 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:09.802067 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.822356 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:09.830051 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:09.833698 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:09.835613 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.849097 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.849684 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.850214 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.862015 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.871331 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.882611 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.889622 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.896593 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.906232 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.907459 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.927778 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:09.932008 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.937588 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:09.940171 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:09.957211 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:09.962538 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:09.965032 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:09.971814 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:09.991684 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:10.044827 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:10.111020 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.111034 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.113286 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.115437 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.126458 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.128651 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:10.128837 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:10.131906 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.153625 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.155259 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.159581 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.160161 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.166708 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.168948 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.200448 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.215090 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.215301 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.219223 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.234417 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.237354 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.242286 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.251717 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:10.252616 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:10.372545 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.382347 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.384406 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.390579 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.392021 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.396000 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.396404 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.410665 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.417397 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:10.417632 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:10.418282 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.436388 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.440434 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.465286 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.472028 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:10.481384 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.492756 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:10.509058 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.517359 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.519222 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.526660 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.530504 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.539726 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.555719 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:10.628688 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:10.640223 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.644601 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.652009 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.652127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.655580 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.657290 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.668792 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.672081 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.680711 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.687335 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:10.693535 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:10.714404 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:10.721822 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:10.724478 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.725978 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.729075 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.733716 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.738747 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.743031 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.748286 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.748956 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:10.760803 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:02:10.763633 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:10.775957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.791810 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.899971 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.904171 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.905766 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:10.922106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.924492 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:10.928128 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.934731 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:10.935878 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:10.949971 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:10.961282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:10.966874 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:10.979521 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:10.983124 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.026896 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:11.043309 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.048713 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.071295 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.092854 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.096515 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.097796 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.103781 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.109616 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:11.115504 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:11.121603 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.126699 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.133517 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.134641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.154847 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.171051 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.186735 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.193722 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.200340 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.200583 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.201638 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.204545 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.212297 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.230052 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.265588 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.332939 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.335327 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.341272 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.344400 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.346494 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.346724 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.356467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:11.397544 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.446453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.448773 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.451643 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:11.456156 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:11.457137 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:11.459775 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.481892 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.481892 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.487002 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.488376 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.497485 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:11.501591 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.503858 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.520587 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.525573 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.526494 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:11.544735 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:11.550650 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:11.570439 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.594318 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.594532 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.602380 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.611227 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.624555 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.624722 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.625726 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:11.669277 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:11.700415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.706465 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.707123 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.713543 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:11.715054 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:11.716775 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.718513 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.730001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.786855 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.791090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.804996 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.808449 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.819247 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.839073 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.839247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:11.845507 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.867542 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.870519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.872145 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.884151 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.901471 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.912331 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:11.913261 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:11.913570 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.915848 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:11.925720 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:11.940693 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:11.943428 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:11.944959 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:11.957149 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.094997 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.096585 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.103173 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.107232 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.107913 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.111040 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.118741 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.123165 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.127351 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.135098 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.137715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.140030 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:12.149621 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:12.157749 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.184600 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.226154 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.226606 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.229853 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.235829 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.245741 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.248109 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:12.265460 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:12.273736 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:12.277729 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.326784 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.330750 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.342478 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.343544 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:12.350467 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:12.359323 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.362605 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.367420 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.417364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.418409 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.421213 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.421593 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.423795 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.433619 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.445084 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.512903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.516270 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.517326 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.519511 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.534649 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.536198 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.562446 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.628679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:12.634281 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:12.640775 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.645282 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.654155 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.666595 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:12.680128 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.681380 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.694698 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.695248 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.697091 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.707476 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.708801 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.714380 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.726478 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.737387 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:12.744768 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:12.786808 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.791527 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.791893 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.792398 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.811444 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.818073 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.824111 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.832226 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(d8994c38d0f640e9bd5b5c322c603b9f): perf score=0.033348
W20240503 08:02:12.836194 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:12.838907 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:12.839305 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6): perf score=0.033350
I20240503 08:02:12.851334 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:12.854080 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:12.877288 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:12.880331 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.884864 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:12.901837 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.911442 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:12.912365 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:12.914229 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:12.946604 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa): perf score=0.033345
W20240503 08:02:12.967229 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:13.029489 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(d8994c38d0f640e9bd5b5c322c603b9f) complete. Timing: real 0.197s user 0.033s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":423,"dirs.queue_time_us":35155,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1136,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3557,"threads_started":3,"wal-append.queue_time_us":37722}
I20240503 08:02:13.031769 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.192s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":47132,"dirs.queue_time_us":33225,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1054,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7393,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":240,"threads_started":3,"wal-append.queue_time_us":17139}
I20240503 08:02:13.033237 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6): 4923 bytes on disk
I20240503 08:02:13.033237 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f): 4838 bytes on disk
I20240503 08:02:13.033910 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.033946 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.034744 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba): perf score=0.033388
I20240503 08:02:13.036597 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(a924487225dc4246aeb695060084096b): perf score=0.033361
I20240503 08:02:13.043424 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.062953 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.070119 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.078105 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.041s user 0.016s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":539,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4570,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":5673}
I20240503 08:02:13.079807 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b): 4923 bytes on disk
I20240503 08:02:13.080216 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(a924487225dc4246aeb695060084096b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.080945 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(d44b2c67014742af9a65a3576f509702): perf score=0.033344
I20240503 08:02:13.099254 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.105350 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.114796 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.119542 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.038s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2951,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4,"lbm_write_time_us":487,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":7914}
I20240503 08:02:13.119917 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.125566 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1): perf score=0.033342
W20240503 08:02:13.168653 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:13.180375 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.044s user 0.016s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":105,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1401,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":6772}
I20240503 08:02:13.181587 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1): 4923 bytes on disk
I20240503 08:02:13.182235 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(20cc823a5ce34916894cfde0a6614bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.185880 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702): 4923 bytes on disk
I20240503 08:02:13.186272 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(d44b2c67014742af9a65a3576f509702) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.197279 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.250s user 0.031s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":9524,"dirs.queue_time_us":48480,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":5653,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8183,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":255,"threads_started":3,"wal-append.queue_time_us":13098}
I20240503 08:02:13.199144 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa): 4838 bytes on disk
I20240503 08:02:13.202643 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.203536 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190): perf score=0.033393
W20240503 08:02:13.209764 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 08:02:13.213544 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.179s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":10049}
I20240503 08:02:13.214545 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba): 4923 bytes on disk
I20240503 08:02:13.215304 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(0e0d84a47cc34af9bdd6b955feb2c3ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.231436 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.027s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":146}
I20240503 08:02:13.232369 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190): 4838 bytes on disk
I20240503 08:02:13.232889 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.281854 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.283816 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.288714 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.291069 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.303228 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.305147 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.325790 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.341257 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.367062 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:13.369675 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:13.373540 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.380506 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.382894 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.388533 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.393544 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:13.409125 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:13.439038 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(c917c5849f54411eb55d69ab33456bf4): perf score=0.033354
I20240503 08:02:13.484706 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1): perf score=0.033397
I20240503 08:02:13.521351 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.036s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":31108,"compiler_manager_pool.run_cpu_time_us":14,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":118,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1293,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":169}
I20240503 08:02:13.522460 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): perf score=0.033383
I20240503 08:02:13.535706 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(c917c5849f54411eb55d69ab33456bf4) complete. Timing: real 0.095s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1345,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":46127}
I20240503 08:02:13.537286 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4): 4838 bytes on disk
I20240503 08:02:13.537678 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.538362 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(70c5de226c7a4b349eef86a79d172851): perf score=0.033361
I20240503 08:02:13.567741 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.045s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":918,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1060,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":28001}
I20240503 08:02:13.568037 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":16139,"compiler_manager_pool.run_cpu_time_us":14,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":828,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1394,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":177}
I20240503 08:02:13.571786 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3): perf score=0.033354
I20240503 08:02:13.572607 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): 4923 bytes on disk
I20240503 08:02:13.573022 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.573735 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(5212dff7ded143a7923bb6176466b0fa): perf score=0.033338
I20240503 08:02:13.597607 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.606842 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.033s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1551,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":8289}
I20240503 08:02:13.608114 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1): 4583 bytes on disk
I20240503 08:02:13.608688 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.609472 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa): 4583 bytes on disk
I20240503 08:02:13.609891 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.611492 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.611574 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.615260 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.043s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1233,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":6401}
I20240503 08:02:13.615706 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.616281 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e): perf score=0.033358
I20240503 08:02:13.628378 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.637392 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.641211 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.687774 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.070s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1957,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":8551}
I20240503 08:02:13.688740 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e): 4838 bytes on disk
I20240503 08:02:13.689673 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.690466 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851): 4583 bytes on disk
I20240503 08:02:13.691385 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.692034 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3): 4583 bytes on disk
I20240503 08:02:13.692363 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.696664 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.700784 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.711539 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.711828 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.712386 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.715109 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:13.720278 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(02d3e4066cd14aa18619310a4490246b): perf score=0.033398
I20240503 08:02:13.722605 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.731381 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.732509 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.740177 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.748559 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.772054 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.052s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1302,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":24005}
I20240503 08:02:13.773005 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d): perf score=0.033403
I20240503 08:02:13.780341 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.785933 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:13.788339 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:13.809585 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:13.816004 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.817019 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:13.822453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:13.826350 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:13.827997 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.831015 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:13.845465 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:13.854786 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.861567 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317): perf score=0.033395
I20240503 08:02:13.867923 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:13.869825 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.097s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67677,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":5734,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":245,"threads_started":2,"wal-append.queue_time_us":79290}
I20240503 08:02:13.870853 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d): 4922 bytes on disk
I20240503 08:02:13.872820 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.873602 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(5212dff7ded143a7923bb6176466b0fa): perf score=0.033416
W20240503 08:02:13.890488 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.893146 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.895036 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:13.896862 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.035s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1129,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":1052}
I20240503 08:02:13.898082 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317): 4583 bytes on disk
I20240503 08:02:13.898543 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.899323 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(a45748a98c3447d3aa2e8de94981a745): perf score=0.033393
I20240503 08:02:13.912907 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.039s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1248,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":484,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1293568,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":13222}
I20240503 08:02:13.914397 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b): 4583 bytes on disk
I20240503 08:02:13.915376 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.916286 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): perf score=0.033417
I20240503 08:02:13.940042 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.041s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1367,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":10581}
I20240503 08:02:13.940042 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.024s user 0.013s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":229}
I20240503 08:02:13.941056 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745): 4838 bytes on disk
I20240503 08:02:13.941493 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.942159 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57): perf score=0.033421
I20240503 08:02:13.943753 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190): perf score=0.033402
I20240503 08:02:13.972265 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.028s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1300,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1251,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":147}
I20240503 08:02:13.973181 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190): 4838 bytes on disk
I20240503 08:02:13.973671 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(5cee7c94dd0941d99d9dd6ceb39fc190) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.995430 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:13.997397 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.054s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9030,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":9688}
I20240503 08:02:13.998314 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc): 4923 bytes on disk
I20240503 08:02:13.999015 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(6ab3fc1d0e6a42f38406bb75f69b8cbc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.999806 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa): 4583 bytes on disk
I20240503 08:02:13.999904 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.000219 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.000892 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57): 4583 bytes on disk
I20240503 08:02:14.001209 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(672fc9981d074f7e8f2324ba0b06be57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.001876 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3): perf score=0.033434
I20240503 08:02:14.020696 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.021157 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.024156 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.027859 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.028312 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.039414 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.056792 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.060024 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.073849 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.072s user 0.007s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":16422}
I20240503 08:02:14.074751 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.074846 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3): 4583 bytes on disk
I20240503 08:02:14.075451 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(d9a2990625964c0e8b79780890b0e6c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.085304 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.086589 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.087519 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:14.179715 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:14.215381 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.218993 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.235494 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.240252 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.248695 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.263423 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.275543 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.280208 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.282987 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.296594 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.296774 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.312556 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.318650 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.319639 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.320317 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.331776 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(9e7f229898c24572815177628720521f): perf score=0.033359
I20240503 08:02:14.357810 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.381570 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.383663 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.392627 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.401027 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(9e7f229898c24572815177628720521f) complete. Timing: real 0.069s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6117,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1849,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":29683}
I20240503 08:02:14.401979 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1): perf score=0.033378
I20240503 08:02:14.403489 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.415665 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:14.416043 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:02:14.425642 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:14.442188 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:14.442625 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:14.444254 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(b094d04434d24923806b0e9e209c3fca): perf score=0.033359
I20240503 08:02:14.445654 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.446460 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.461912 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":242,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 08:02:14.462906 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca): 4923 bytes on disk
I20240503 08:02:14.463157 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.463675 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(b094d04434d24923806b0e9e209c3fca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.469885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.473848 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.475042 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.479930 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.480345 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d): perf score=0.033365
I20240503 08:02:14.500159 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.098s user 0.012s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1495,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":62659}
I20240503 08:02:14.501320 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1): 4583 bytes on disk
I20240503 08:02:14.501780 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(1c84f7a57e0945d9a25a70798f054bd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.502619 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f): 4581 bytes on disk
I20240503 08:02:14.502971 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.521320 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.041s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4116,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1426,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":17716}
I20240503 08:02:14.522315 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d): 4922 bytes on disk
I20240503 08:02:14.523119 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(a60b868dcc504bb29d08e2752c8dd44d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.549463 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.554414 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.557420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.563727 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.574626 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.577505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.606396 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:14.616588 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:14.618770 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.627821 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.628242 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.636620 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.644678 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.651777 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.668278 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.715150 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45): perf score=0.033401
I20240503 08:02:14.743793 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.028s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1139,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":1420,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1408,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":3240}
I20240503 08:02:14.744684 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(f1151731adc9407b86d143d6d41f69b4): perf score=0.033385
I20240503 08:02:14.754345 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(5480e544dca3428ab4fcf190577d5331): perf score=0.033400
I20240503 08:02:14.771023 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.771927 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.775960 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(5480e544dca3428ab4fcf190577d5331): perf score=0.033363
I20240503 08:02:14.783759 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.784024 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.802513 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.804903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.805939 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:14.808188 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:02:14.810137 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:14.810024 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.055s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1249,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":13799}
I20240503 08:02:14.817871 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331): 4836 bytes on disk
I20240503 08:02:14.818468 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.818710 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.819691 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.829393 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.835891 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.841655 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.856351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:14.882393 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.133s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":761,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1059,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":16427,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":60727}
I20240503 08:02:14.883289 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.883423 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45): 4838 bytes on disk
I20240503 08:02:14.884107 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.884882 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4): 4838 bytes on disk
I20240503 08:02:14.885408 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(f1151731adc9407b86d143d6d41f69b4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.886085 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(7d61aa445a50417b82916019c2a4298c): perf score=0.033423
W20240503 08:02:14.892822 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:14.905699 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:14.922732 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.147s user 0.013s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":808,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1108,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":1092}
I20240503 08:02:14.923638 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331): 4836 bytes on disk
I20240503 08:02:14.924175 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(5480e544dca3428ab4fcf190577d5331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.928385 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:14.929489 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:14.930119 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:14.941711 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:14.954274 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.961763 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:14.980412 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:14.987473 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.000871 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.115s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1168,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":94748}
I20240503 08:02:15.001793 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c): 4923 bytes on disk
I20240503 08:02:15.002254 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(7d61aa445a50417b82916019c2a4298c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.041252 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.043062 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.054282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.055146 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.055706 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.062325 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.063350 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:15.125262 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:15.135533 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.148746 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.150638 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.151533 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.159124 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.169588 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.175793 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling FlushMRSOp(5212dff7ded143a7923bb6176466b0fa): perf score=0.033376
W20240503 08:02:15.181784 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.182941 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.213274 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: FlushMRSOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.037s user 0.014s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18076,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1656,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":878}
I20240503 08:02:15.215621 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa): 4583 bytes on disk
I20240503 08:02:15.217994 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(5212dff7ded143a7923bb6176466b0fa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.219741 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.223012 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:15.225631 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.226135 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.233316 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.236220 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.248668 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.262128 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.356822 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.360805 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.370890 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.377785 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.382835 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.386174 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.387004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:15.457568 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.467164 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.476843 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.480587 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:15.483371 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:15.488418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.490464 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:15.494020 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:15.495193 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:15.497089 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.498481 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.512408 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.516494 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.520090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.527767 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.553720 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.557312 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.559947 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.609107 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.613373 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.619292 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.620720 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.624115 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.626578 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.631975 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.638206 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.648209 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.675170 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.683516 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.685798 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.685925 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.695248 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:15.731647 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:15.807390 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.810029 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.810214 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.815243 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.821581 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.831552 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:15.844138 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.844215 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.844525 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:15.859733 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:15.869361 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.870373 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:15.874086 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:15.877952 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.896478 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.948282 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:15.949875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:15.954865 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:15.966104 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:15.967897 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:15.969099 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:15.979967 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:15.990970 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.014218 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.049937 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.055397 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.056502 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.061502 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.075073 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.079556 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.088217 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.138123 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.141228 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.146440 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.146528 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.153192 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.163100 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.209851 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:16.244096 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:16.276608 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.283058 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.288594 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.291559 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.292644 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.311071 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.323917 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.343681 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.360783 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.364836 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.369066 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.373909 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:16.383656 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:16.389624 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.390760 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.395176 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.397590 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.398849 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.402899 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.403991 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:16.407999 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:16.420500 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.435066 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:16.459803 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:16.473875 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:16.535130 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.540083 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.541999 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.543825 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.545936 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.547223 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.594746 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:16.600785 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 08:02:16.601657 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:16.605175 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:16.612203 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:16.616573 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.617590 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.617803 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.627351 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.643733 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.657815 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.660672 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.665359 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.670156 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.673753 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.681582 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.714426 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.715842 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.749325 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:16.750517 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.753650 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.753902 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.757237 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.759045 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.773662 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.861511 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:16.864629 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:16.865903 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.869652 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:16.873903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.876873 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:16.877105 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.909968 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.940037 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:16.942437 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:16.947830 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:16.957335 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:16.988226 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.991420 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:16.991561 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:17.013233 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:17.021704 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.041111 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.071177 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.071635 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.071883 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.083223 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.083279 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.083590 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.114122 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.117817 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.119704 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.121685 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.134735 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.137663 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.153798 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:17.160763 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:17.206341 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:17.256371 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.258775 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.264149 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.267758 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.274304 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:17.285230 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.289104 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:17.298342 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.298547 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.358453 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.372548 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.375883 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.379465 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.391858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.397367 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.397418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.397789 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.399878 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.405786 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.416462 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:17.428795 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:17.429632 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.449306 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.457260 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:17.521521 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:17.535413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.541349 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.545924 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.558319 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.559816 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.562769 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.581561 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.636745 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.639994 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.644877 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.650581 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.651789 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:17.667176 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:17.674340 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.677814 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.687501 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.688817 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.700866 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.701891 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.704383 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.727108 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:17.737694 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:17.739782 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:17.751497 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:17.767650 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.770534 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.770596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.778595 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.785077 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.787777 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.796619 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.894229 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.897043 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.915051 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.920943 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.929735 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:17.948005 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:17.948065 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:17.969050 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:17.977826 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:17.979270 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:02:17.986680 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:17.987243 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:17.990336 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:17.999315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:18.009500 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:18.020004 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.020134 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.021420 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.025485 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.027474 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.031116 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.059021 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.062819 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.067734 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.191304 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.201810 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.208217 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.209038 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.216398 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.238984 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.254591 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.265129 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.270303 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.278460 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.284118 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.286815 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.293694 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.296875 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.306421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:18.317219 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:18.318550 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.321460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.324461 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.338845 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(076781af2ac14ddc87d31e3ec77db08f): perf score=0.033359
I20240503 08:02:18.347604 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.359969 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.365639 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.411262 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.416008 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.417140 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.424404 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.426631 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(076781af2ac14ddc87d31e3ec77db08f) complete. Timing: real 0.084s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5793,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":2322,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":33375,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":24381}
I20240503 08:02:18.427670 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f): 4582 bytes on disk
I20240503 08:02:18.428097 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 08:02:18.442160 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.446681 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:18.463104 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:18.467723 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:18.496256 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:18.528251 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:18.554154 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.560042 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.560326 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.573508 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.579203 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.582420 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.596072 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.596529 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.597704 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.599777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.607249 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.609331 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:18.633766 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:18.635368 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.643069 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.705148 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.711509 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.713833 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.727783 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.729889 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.734227 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.745934 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.824343 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:18.839308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:18.839813 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:18.853094 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:18.853768 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:18.855517 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.860337 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.861450 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:18.866235 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.878086 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:18.879590 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:18.883301 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.883687 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:18.892704 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:18.896390 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:18.899816 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:18.910238 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:18.975447 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:19.019716 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:19.024230 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:19.029661 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.043377 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.046725 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.071066 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:19.086355 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:19.091212 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.103384 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.105898 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.114717 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.122440 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.131580 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.136516 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.141618 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.173857 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.177412 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.183713 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.187096 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.190644 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.196781 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.216668 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.219913 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.269954 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.275169 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.275427 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.287166 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.290938 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.315618 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.335083 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.408109 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.410398 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:19.412719 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:19.424373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.427964 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.431895 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.440157 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.443820 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.450037 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.450109 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.487313 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.491824 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.499783 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.508626 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.527452 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.591202 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.596757 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.605011 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:19.606657 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:02:19.607543 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:19.611894 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.616312 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.618542 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:19.632287 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:19.646232 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.657543 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.666191 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.668524 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:19.679227 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:19.681196 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.696920 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.704808 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.705148 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:19.748358 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:19.764830 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.774231 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.775449 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.776021 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.778724 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.791376 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.813581 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.816574 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.816951 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.817785 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.823896 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:19.826155 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:19.838742 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.839167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:19.844275 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:19.845949 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.853734 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:19.867446 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:19.869911 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:19.877550 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.895534 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:19.918462 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.000953 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.001777 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.002211 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.010309 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:20.012841 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:20.028749 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:20.050817 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:20.051874 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:20.052143 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.060545 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.109333 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.110172 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.114821 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.117549 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.128659 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.151508 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.151963 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.152814 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.175125 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.177426 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.185190 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.198091 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.201562 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:20.205130 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:20.207692 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:20.245450 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:20.324018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.332001 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.343979 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.352712 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.359568 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.360402 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.361924 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.374994 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.382541 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.391232 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.391279 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.416766 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.433655 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.434218 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:20.466267 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.473835 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:20.475217 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.477334 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.481496 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.482514 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.482822 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.484544 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.490779 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:20.537442 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:20.552139 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.558063 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.559376 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.562156 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:20.562543 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:20.567201 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.592243 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.595113 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.599817 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.617534 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.621798 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.629158 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.641803 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.642589 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.649403 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:20.657217 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:20.720641 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.724486 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.744375 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.748968 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.764570 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.774717 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.783213 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.784919 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.797088 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.816751 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.824542 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:20.831451 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:20.831574 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.839113 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.839371 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:20.856611 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.857102 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:20.865388 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:20.867599 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.887542 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:20.891083 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.006889 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.008942 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.015120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:21.024039 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:21.043838 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.044466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.048957 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.055476 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.106395 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.117851 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.128715 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.129207 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.133810 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.147795 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.161404 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.163636 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.164268 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.175508 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.178566 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.189677 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:21.190871 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:21.199416 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:21.204825 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:21.217003 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.303632 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.320641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.322070 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.325585 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.332369 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.341390 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:21.344602 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:21.345335 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:21.370488 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.386142 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:21.401856 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:21.402375 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.403128 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.411885 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.413571 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.423955 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.441044 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.445106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.460872 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.462272 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.464072 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.464931 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.465574 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.497104 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.501667 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.502530 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.503602 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:21.520267 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:21.521363 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.534547 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.546638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.559455 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:21.564669 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:21.570562 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:21.615595 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:21.654307 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:21.694275 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.701036 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.709429 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.714197 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.722136 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.722687 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.735106 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.736670 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.754595 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.758567 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.759449 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.777165 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.780465 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.810622 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.821754 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:21.830235 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.832243 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.840564 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.844400 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.846292 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.847589 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.849643 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.851454 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:21.860641 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:21.875666 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:21.877192 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:21.880595 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:21.885510 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:21.919862 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:21.981644 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.984485 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.985291 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:22.046612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.052469 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.070753 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.071630 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:22.077406 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:22.079600 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:22.084584 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.085840 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.087350 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.088478 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.092154 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.100916 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.119797 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.130244 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.142135 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.142808 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:22.162861 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:22.182971 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.183710 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.191928 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.193156 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.199729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.214063 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.228390 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.239946 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.240449 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.242607 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.244786 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.247804 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.259649 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.289963 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.362315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.368806 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.372152 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.377976 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.379817 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.381747 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.406669 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:22.432456 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:22.450825 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.453697 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.455956 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.465515 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.469569 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.476886 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:22.492394 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:22.503226 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.515801 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.516276 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.521407 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.526544 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.530465 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.531369 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.550387 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:22.621220 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:22.651134 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.654891 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.668846 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.670599 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.681092 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.690783 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.703860 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.703893 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.718410 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.719116 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.719323 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.727584 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.734710 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:22.734815 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:22.740217 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.741950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.746405 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.767088 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.780736 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.783200 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.783711 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:22.784158 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.877861 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.878710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.904166 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:22.917567 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:22.923911 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.924798 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.933961 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:22.944556 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.961531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:22.963483 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:22.980553 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:22.980814 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:22.981787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.004299 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.018700 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.030165 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:23.034592 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 08:02:23.034950 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:23.038182 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:23.040710 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.047466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.051833 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.057498 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.065651 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.072726 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:23.113209 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:23.141554 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.141554 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.144544 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.164230 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.166533 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.178815 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.192113 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.261695 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.262356 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.271615 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.278534 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.280927 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.281837 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.296931 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.301751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.322638 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.323295 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.324517 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.329762 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.335935 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.356366 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.438653 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.440411 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.445000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.471411 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.473636 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.474045 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.474720 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.477885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.489378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.490036 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.504875 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.507164 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:23.521332 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:23.523816 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.524073 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:23.532364 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:23.537462 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.537706 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.546751 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.547400 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.571434 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.580403 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.588585 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.692687 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.702473 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.708127 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.746902 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.751520 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.762862 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.766026 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.766631 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:23.776549 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:23.777091 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.780689 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.787887 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling FlushMRSOp(48e4f2a993164fb6b1df674311698006): perf score=0.033372
I20240503 08:02:23.789655 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.799350 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:23.801734 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.805418 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.811106 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.811087 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.811311 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.822693 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:23.836629 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:23.837920 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:23.843569 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.871697 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: FlushMRSOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.084s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":177,"threads_started":2,"wal-append.queue_time_us":9485}
I20240503 08:02:23.873018 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006): 4584 bytes on disk
I20240503 08:02:23.873533 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:23.879388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:23.896382 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:23.903069 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:23.917454 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:23.923395 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:23.923982 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:23.926013 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:23.926604 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.936138 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:23.936686 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:23.945058 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:23.945827 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.951647 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:23.986055 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.001376 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.002000 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.003304 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.008802 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.015524 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.024194 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.043593 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.107515 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.112993 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.114197 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.123953 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.126107 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.126214 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.127104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.205633 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.206558 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:24.218894 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 08:02:24.227370 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.228262 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.249698 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.266026 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.275740 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.288633 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.292606 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.305815 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.314894 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.327576 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.328887 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.362632 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.368717 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.372189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.378266 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.382081 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.395517 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.416115 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.429968 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:24.433142 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:24.433205 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.437436 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.440039 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.441455 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.451938 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.455699 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:24.455999 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:24.465385 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.468288 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa): perf score=0.033343
W20240503 08:02:24.472831 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:24.497716 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10793,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":2631,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":190,"threads_started":2,"wal-append.queue_time_us":198}
I20240503 08:02:24.498675 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa): 4838 bytes on disk
I20240503 08:02:24.499223 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(9c7bf40aa0614923a19feed8bab430fa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 08:02:24.502079 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:24.569324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.582393 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.584646 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.585913 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.587090 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:24.600965 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:24.624166 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.629220 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.645519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.650877 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.656668 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.659682 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.663033 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.665457 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.666114 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.667120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.685259 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.686102 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.699581 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45): perf score=0.033347
I20240503 08:02:24.721000 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.722108 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: FlushMRSOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":2207,"drs_written":1,"lbm_read_time_us":2396,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":138}
I20240503 08:02:24.722963 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45): 4838 bytes on disk
I20240503 08:02:24.723722 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(72ac9fbe5dfd4a4e89912d7e947eca45) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:02:24.729990 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.737689 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.758183 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:24.838169 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:24.841017 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.842599 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.847469 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.854032 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.866818 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.868544 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.870760 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:24.874642 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.875399 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.876576 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.880795 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:24.883371 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 08:02:24.886772 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:24.900995 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:24.905344 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.914688 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:24.979439 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:24.980042 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:24.984416 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:24.984778 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:24.998369 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.001441 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317): perf score=0.033360
I20240503 08:02:25.021414 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.023473 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:25.056445 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:25.064935 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.063s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21257,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2274,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19944,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":164}
I20240503 08:02:25.067307 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317): 4583 bytes on disk
I20240503 08:02:25.068347 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240503 08:02:25.098802 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.107607 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.108186 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.122401 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.129040 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.129446 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:25.154177 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:25.163012 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.172813 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.175949 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.195255 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.195617 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.209115 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.209414 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.244477 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.252403 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.261454 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.262799 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.265033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.269594 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.303393 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.307626 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.317067 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.332693 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.332748 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.336181 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.344903 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.351675 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:25.353977 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:25.359014 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.368875 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.375280 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.463716 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.475915 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.478749 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:25.479950 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:25.483405 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.483556 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:25.493793 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.495196 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.505591 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.557057 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.571861 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.575377 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.579270 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:25.579308 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:02:25.607887 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.611550 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.616891 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.619292 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.620049 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:25.627532 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:25.634299 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.634378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.645629 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.647423 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6): perf score=0.033393
I20240503 08:02:25.658150 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.658959 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.672724 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.673136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.681529 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.683274 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.703783 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.056s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9468,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":511,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":7407}
I20240503 08:02:25.707371 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6): 4923 bytes on disk
I20240503 08:02:25.707846 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(dcdde0a31f7d4fd0bf20c894f215e3b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:25.718034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.734280 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.739696 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.880393 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.882750 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.889307 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.897284 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.897315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.902196 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:25.917649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.919304 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.921353 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:25.925027 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:25.930416 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:25.931152 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:25.943588 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.946215 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.957263 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:25.959233 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317): perf score=0.033349
I20240503 08:02:25.971834 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.973141 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:25.973325 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:25.993098 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.013684 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.022207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.023562 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.032691 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.035331 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.036593 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.044083 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.068125 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.069201 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.077085 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:26.084286 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:26.103574 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: FlushMRSOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.144s user 0.015s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":100,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1988,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":67798,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2944,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":152}
I20240503 08:02:26.104645 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317): 4583 bytes on disk
I20240503 08:02:26.105161 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(7c630e5cf7c0419cbf5d13413d22d317) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:26.188690 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.188946 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.190351 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.191869 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.201581 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.203995 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.217341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:26.224841 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:26.262413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.266793 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:26.270169 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:26.271023 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.271492 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.280140 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.282227 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.284910 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.325572 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e): perf score=0.033368
W20240503 08:02:26.332805 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:26.342242 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4169,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":251}
I20240503 08:02:26.343192 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e): 4838 bytes on disk
I20240503 08:02:26.343853 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(389b40c2136d45ab86ccf49df8d1b71e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 08:02:26.398663 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.405364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.409385 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:26.418064 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:26.427297 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.431754 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.449616 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:26.451447 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:26.453284 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:26.463651 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:26.487951 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:26.495082 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.496098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.505390 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.506206 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.515573 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.517207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.538157 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.543963 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.545137 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.549896 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.557915 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.561046 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.563498 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:26.584334 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:26.595464 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(70c5de226c7a4b349eef86a79d172851): perf score=0.033381
I20240503 08:02:26.621484 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.640708 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.641127 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.648000 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.648769 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.657769 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.659485 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.660193 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.663700 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.068s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":2525,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2000,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":129,"threads_started":1,"wal-append.queue_time_us":6331}
I20240503 08:02:26.664662 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(02d3e4066cd14aa18619310a4490246b): perf score=0.033394
I20240503 08:02:26.666532 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.668944 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.684250 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:26.689563 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:02:26.692752 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.028s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":167}
I20240503 08:02:26.693814 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b): 4583 bytes on disk
I20240503 08:02:26.694191 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(02d3e4066cd14aa18619310a4490246b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:02:26.694952 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851): 4583 bytes on disk
I20240503 08:02:26.695263 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(70c5de226c7a4b349eef86a79d172851) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 08:02:26.698019 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.713176 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling FlushMRSOp(48e4f2a993164fb6b1df674311698006): perf score=0.033368
I20240503 08:02:26.724499 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.730826 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:26.738710 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.744926 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: FlushMRSOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.032s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5757,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":3523,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":10213}
I20240503 08:02:26.745813 14705 maintenance_manager.cc:419] P 8398e6f89bf5419d81ed5bed991da0cf: Scheduling UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006): 4584 bytes on disk
I20240503 08:02:26.746203 14595 maintenance_manager.cc:643] P 8398e6f89bf5419d81ed5bed991da0cf: UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 08:02:26.817826 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:26.819449 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:26.820915 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.822909 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.824433 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.827539 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.829406 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.832233 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.900839 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.904316 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:26.905858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:26.906272 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:26.931327 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:26.933415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:26.934062 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:26.941836 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:26.992005 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:26.995751 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:26.999111 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.005072 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.007620 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.025863 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.047076 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.057355 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:27.067379 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:27.102722 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.106729 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.117926 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.123318 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.123559 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.124853 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.136781 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:27.144881 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:27.194380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.204717 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.206037 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.209407 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.211787 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.247476 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.257246 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:27.273214 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:27.322031 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.325361 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.328411 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.329663 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.332290 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.337704 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.338985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.370824 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.371834 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:27.374869 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:27.376566 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.384174 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.384774 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.390386 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.407156 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.436930 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.443302 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.449605 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.451965 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.469731 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.471415 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:27.476114 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:27.481323 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:27.501976 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:27.612551 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.615373 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.618010 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.625083 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.628171 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:27.638118 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:27.643529 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.643851 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.649884 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.668053 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.670107 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.675521 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.692159 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.702453 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.716719 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.737370 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:27.738229 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:27.739965 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:27.748273 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:27.750238 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.758348 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.760778 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.777938 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.778453 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.783398 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.786293 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.794036 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.799661 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.801028 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.801445 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.804478 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.908071 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:27.910300 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.920986 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:27.926048 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:27.937042 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:27.937247 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:27.939790 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:27.950212 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47): perf score=0.033361
W20240503 08:02:27.986815 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:27.992653 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: FlushMRSOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.042s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2223,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":24012,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":146}
I20240503 08:02:27.993691 12993 maintenance_manager.cc:419] P 0cf63591cfc244028e1fc368c2bca079: Scheduling UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47): 4838 bytes on disk
I20240503 08:02:27.994132 12883 maintenance_manager.cc:643] P 0cf63591cfc244028e1fc368c2bca079: UndoDeltaBlockGCOp(d376ac1ea05c482ca8f03ddf0f5fcd47) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 08:02:28.035118 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:28.043978 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.051720 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.052189 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.052824 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:28.056228 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:02:28.056743 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.070843 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.111752 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.128134 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.129426 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.138069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.140223 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.140988 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.167600 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.170166 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.187945 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.200345 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.229635 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.230099 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.238798 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.254379 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.256127 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.258827 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.260664 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.269061 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.270210 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.271366 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.300563 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.301054 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:28.339087 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:28.391278 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:28.412251 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.419878 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.423827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.432055 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.434815 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.439599 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.459640 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.464624 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.466270 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.475104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.481489 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.489260 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.504272 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.546159 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.546756 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.548591 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.553557 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.565030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.568046 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.576565 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.579607 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.647087 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.655073 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.658556 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.659286 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.669793 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.671291 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.700428 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.738277 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.746666 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:28.748545 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:28.749636 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.754961 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.757812 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.779558 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.781145 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.783715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:28.784759 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:28.790395 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.799558 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.806447 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.813344 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:28.836020 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:28.859155 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:28.868973 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.890138 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:28.890837 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:28.891307 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:28.894395 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:28.895108 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:28.903982 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:28.922894 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:28.927482 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:28.940205 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:28.957732 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.000821 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.009168 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.015687 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.017293 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.024435 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.028223 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.054809 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:29.066385 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:29.079120 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.079906 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.085201 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.085606 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.091423 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.101500 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.126317 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:29.134863 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:29.211643 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.217232 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.219507 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.222510 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.230448 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.234005 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.235313 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.242231 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.244724 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.254959 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.256934 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.260339 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.270341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.277611 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:29.357550 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.373198 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:29.409579 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:29.442413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.453199 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.453503 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.455024 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:29.483187 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:29.486984 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.488549 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.494458 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.495842 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.503005 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.505777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.506639 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.508209 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.508307 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.527714 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.529299 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.544618 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.547406 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.559093 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.559116 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.565513 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.575201 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:29.589006 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:29.649756 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.652019 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.653259 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.659382 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.660764 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.660985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.677598 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:29.705824 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:29.761366 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.774821 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.786696 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.795902 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.807545 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.808744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.809798 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.812866 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.815632 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.824651 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.834427 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.843214 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.846171 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.863902 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:29.872562 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:29.925457 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:29.926127 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:29.930802 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:29.931972 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:29.939172 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:29.939608 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:29.949821 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:29.950078 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:30.005314 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:30.035457 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.043277 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.044052 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.055801 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.057590 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.064710 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.073884 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:30.091912 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:02:30.093217 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:30.134435 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.138687 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.139465 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.153506 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.154480 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.158226 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.159816 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:30.166018 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:30.179430 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.181080 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.188964 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.191256 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.194329 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.202283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.203975 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:30.302659 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:30.304400 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:30.343888 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.348549 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.354416 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.360924 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.365725 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.370442 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.371210 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.382831 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.383867 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.411250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.412276 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.420477 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.450325 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.461086 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.466849 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.474822 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.477183 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.478457 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.484454 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.490984 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.496981 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:30.520675 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:30.562394 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:02:30.565560 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:30.595008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.595253 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.602591 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.615283 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.618119 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.621855 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.628284 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:30.669327 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:30.690416 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.700364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:30.702014 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:30.710049 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.712440 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.713631 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.714109 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.726063 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.742312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.754832 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.755168 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.765841 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.780550 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.784163 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.793403 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.832464 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.837124 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.847399 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.859777 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:30.859766 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:30.871865 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.889441 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:30.971365 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:30.978226 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:30.997366 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:30.999022 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.003639 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.016863 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.023918 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:31.026144 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:31.029024 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.029285 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.031322 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.047904 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.048547 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.049787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.070068 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.072892 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.075325 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.085001 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.086659 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.088572 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.097695 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.123466 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:31.169525 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:31.247560 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.253803 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.260411 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.261622 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.263670 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:31.263721 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:31.265350 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:31.273018 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:31.279817 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.290691 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.293936 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.295256 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.318261 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.332010 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.348013 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.367612 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.381997 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.383311 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.386639 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.390470 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.391690 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:31.401590 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:31.411064 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.416962 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:31.491988 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:31.528774 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:31.536512 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.536516 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.550802 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.553606 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.585060 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.592454 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.598166 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.625686 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.627542 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:31.634721 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.635156 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:31.645578 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:31.649214 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.667757 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.669656 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.672418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.685388 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.689083 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.697548 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.705381 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.718169 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.719280 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.720631 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:31.724131 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:31.731176 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.731789 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.732287 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.734279 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.748726 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.751164 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:31.786547 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:31.832509 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.835072 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.836093 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.836956 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.843920 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.847959 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.860354 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:31.887324 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:31.942234 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:31.947100 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:31.950182 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:31.950217 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:31.951524 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:31.962998 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:31.974371 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:31.975512 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:31.983544 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:31.984512 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:32.025154 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.029696 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.033280 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.040125 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.057328 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.060894 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.076694 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:32.110260 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:32.127193 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:32.189945 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.203984 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.222702 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.230698 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.237126 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.245751 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.246623 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.250309 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.252591 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:32.257464 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:32.257740 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.259547 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.274122 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.285048 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.287178 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.295739 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.298004 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.304790 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.314988 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.327970 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:32.332757 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:32.336894 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.337954 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:32.339859 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:32.342810 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.347906 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.351004 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.352854 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.354861 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.356362 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.379055 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:32.522624 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:32.547003 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.551607 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.551606 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.551807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.552723 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.577245 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:32.587599 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:32.592780 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:32.606442 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:32.618628 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.618857 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.622834 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.641140 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.651664 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.656534 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.663341 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.664533 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.672355 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.673293 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.689288 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.692093 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:32.694674 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:32.713591 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.714215 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.736599 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.740521 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.755895 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.759115 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.767742 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.773476 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.785763 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.870621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.870977 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.879364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.891731 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:32.899870 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.908396 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.909832 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:32.961522 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.961829 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:32.963693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:32.975773 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:32.977102 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:32.978422 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.022859 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.043489 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.050819 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.053130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:33.061810 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:02:33.074076 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.078088 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:33.078495 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:33.079834 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.081786 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.084002 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.089121 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.099432 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.103049 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.103375 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:33.105530 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:33.118070 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.125059 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:33.212276 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.235193 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:33.290848 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.292302 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.309178 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.316076 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.321600 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.333885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.343375 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.347561 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.348382 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.348616 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.351764 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.366506 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.371479 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.387943 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.391808 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.397866 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.408478 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.414301 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.422058 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.441952 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.471434 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.508217 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.519248 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.520148 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.521651 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.525576 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.547122 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.564065 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:33.591302 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:33.611663 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.615797 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.616130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.619071 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.624325 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.639178 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.654361 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:33.657526 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:33.663734 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.668146 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.682324 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.709692 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.715704 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.728583 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.729143 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:33.777406 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:33.799772 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.806664 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.813037 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.824986 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.825274 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.825959 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.827651 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:33.834437 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:33.884708 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:33.892493 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.893611 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.894484 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:33.894663 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 08:02:33.899354 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.904300 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:33.935428 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:33.946117 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:33.954851 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:33.959813 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:33.979867 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.002274 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.007429 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.032445 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:34.040746 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:34.070268 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:34.078619 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.080271 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.088589 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.095515 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.097695 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.103127 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.128304 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.157593 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.189302 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.196040 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.199031 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.200078 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.225358 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:34.226691 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:34.227681 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:34.241930 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.247262 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.267868 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.274050 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.281641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.285955 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.292393 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.301353 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.380654 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.381889 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.382042 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.382403 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.384601 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.409140 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.419648 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.441139 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.441792 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.442816 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.446107 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:34.459981 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:34.469761 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.483239 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.484459 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.518796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.521714 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.523546 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:34.532771 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:34.538401 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.558019 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.558517 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.564081 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:34.622081 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:34.629643 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.636593 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.653049 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.660040 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:34.664486 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:34.669786 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:34.673326 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:34.678294 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:34.679796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.680574 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.688023 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.704545 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.704959 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.706363 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.719837 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:34.741367 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:34.751159 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:34.753453 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:34.753736 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.755558 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.765925 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.774752 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.778075 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.787096 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.788725 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.810500 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.907579 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:34.911392 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.918773 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.923321 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.923578 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:34.926743 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:34.929013 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:34.940371 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:34.965790 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:34.979094 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:34.979465 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:34.986136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.000967 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.019109 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.021448 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.098551 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.114035 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.115366 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.125458 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.136023 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.139766 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.143599 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.199502 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.230428 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.232553 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.233702 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.233803 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:35.236819 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 08:02:35.245191 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.260779 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:35.310374 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:35.320832 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.322189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.330864 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.331739 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:35.334450 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:35.335096 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:35.347039 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.355976 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.370519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.406845 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.407804 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.420128 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.422688 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.428119 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.437028 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.440644 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.492839 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.494748 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.497737 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:35.505148 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:35.512821 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.513904 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.515120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.518093 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:35.566149 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:35.644762 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.649274 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.652462 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.660269 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.662218 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.665104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.676367 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.678704 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.687662 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.700337 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.721875 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:35.732360 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:35.738337 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:35.741276 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:35.742789 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.746601 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:35.752089 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:02:35.754253 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:35.754880 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.756934 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.759995 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.764688 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.768421 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.782771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.785496 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.788097 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.794629 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.815744 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:35.832298 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.857836 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.858130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.858902 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.916555 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.916656 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:35.922614 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:35.922864 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:35.926079 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:35.928354 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:35.946362 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.032430 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.036038 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.036581 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.045334 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.047534 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.054291 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.057955 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:36.129253 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:36.193120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.195086 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.204119 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.216923 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.224344 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:36.233466 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:36.234344 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:36.251029 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:36.251442 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.271492 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.282828 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.283186 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.286804 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.288226 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.337920 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.343125 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.349676 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.350625 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.350860 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.352032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.353056 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.362124 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.368185 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.378657 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.382045 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:36.400972 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:36.408325 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.411521 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.422657 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.431670 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.439611 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:36.508836 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 08:02:36.531235 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.536362 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.536867 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:36.537195 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:36.544880 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.550029 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.550557 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.567332 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.605412 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.613476 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.618117 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.622602 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.647531 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.648557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.648557 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.673889 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.677067 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.686415 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:36.687932 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:36.692643 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:36.702035 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:36.704583 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.704993 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.709146 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:36.729048 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.744809 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:36.831575 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.832823 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.841885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.853020 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.853111 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.865481 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.870122 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.871271 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.881799 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.895300 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.905656 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:36.912657 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.927043 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.933598 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:36.941566 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:36.950497 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:36.954213 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.954591 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:36.974172 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:36.993505 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.039790 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:37.059644 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:02:37.060341 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:37.067873 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.069092 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.071875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.071895 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.084017 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.084105 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.089231 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:37.107667 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:37.115795 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.163637 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:37.167088 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.170097 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.174495 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:37.181716 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:37.189221 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.189786 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.190562 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.198619 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:37.229951 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:37.264626 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.265059 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.265540 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.271739 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.279664 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.286116 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.293088 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.441424 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:37.458580 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 08:02:37.469594 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.470127 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.483898 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.484607 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.488541 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.489307 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.501353 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.501466 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.506457 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.513175 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:37.530812 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:37.542300 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.543130 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.550172 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.561892 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.565798 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.577275 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.577873 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.581125 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.585033 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:37.599923 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:37.606751 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:37.640302 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:37.661209 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.666821 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.670475 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.674176 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.680572 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.686115 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.687667 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.701304 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.715930 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.718644 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.726212 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.732477 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.752367 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.752841 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:37.825863 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:37.833609 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.833851 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.841742 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.843468 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:37.846997 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.871228 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.883641 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.909895 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:37.921360 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.922823 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:37.926103 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:37.927093 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:37.929811 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:37.944216 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:37.965129 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.031142 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:38.057586 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:38.071055 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:02:38.078433 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.086794 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.091862 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:38.101459 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:38.111563 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.117789 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.119096 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:38.148231 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:38.165575 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.166227 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.166678 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.171298 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.183070 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.189050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.191171 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.241331 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.255710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.271852 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.271852 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.284466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.287755 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.289816 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.290990 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.308264 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.323916 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.325011 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.329725 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.334444 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.358430 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.390415 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.394347 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.410003 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.414077 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.417543 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.418283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.419039 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:38.510195 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:38.524600 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.545502 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.546078 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.548230 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.549738 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.564761 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.578274 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.580897 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.587098 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.594702 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.602058 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.610388 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.627897 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:38.646641 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:38.654822 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.688491 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.689292 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.699236 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:38.713702 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:38.714394 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.725885 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.734869 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.734869 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.746397 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.751202 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.757565 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.769654 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.773264 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.788969 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.798692 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.882740 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.887089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.892161 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.893689 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:38.897363 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.898126 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.900601 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.902489 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:38.916728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:38.919503 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:38.938557 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.945734 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:38.977160 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:38.980015 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:39.011307 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:39.031457 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:39.097208 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:39.103156 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.106359 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.118719 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.122613 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:39.128582 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.130553 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.135687 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.140115 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.148335 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.149412 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.153205 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.171123 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:39.172999 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.175316 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.208189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.217582 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:39.249541 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:39.263413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.263823 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.273331 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.291491 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.294677 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.310488 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.333948 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.339147 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.343797 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.351146 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.355988 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.360380 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.373873 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.379011 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:39.397215 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.415910 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.422739 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.425367 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.432705 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.434621 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.450839 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.454990 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:39.537961 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.542397 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.542752 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.551252 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.552929 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.557300 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.567523 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:39.591166 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:39.593447 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:39.608579 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.626463 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:39.639178 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:39.641064 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.643085 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.652827 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.664871 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:39.674301 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:39.675752 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.693661 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.697744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.709126 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.711187 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.723676 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.729212 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:39.731513 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 08:02:39.733999 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.739589 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.740633 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.756471 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.770423 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.783672 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.789054 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.789266 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.791141 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.818346 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.882325 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:39.889321 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.899544 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:39.899853 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:39.900934 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:39.921679 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:39.934062 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:39.997848 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.005841 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.006114 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.012091 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.022158 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.028494 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.029983 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.041000 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.054014 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.058755 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.063525 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.067615 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:40.077543 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:40.093462 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.108326 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:40.122666 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:40.165603 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:40.208240 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.208251 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:40.210258 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:40.222129 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.222375 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.226655 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.264289 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.267597 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.320585 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.320597 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.332995 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.344472 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.354519 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.360239 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.360219 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:40.375847 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:40.384629 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.387521 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.393906 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.403007 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.405022 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.405741 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.426707 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:40.443821 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 08:02:40.445502 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.456249 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.456928 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.458427 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.461230 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.466253 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.474846 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.476958 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.478911 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.504168 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.518015 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:40.525835 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:40.539584 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.541047 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.575877 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:40.578518 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:40.656700 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.663307 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.663422 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.676301 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.687496 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.690667 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.695112 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.704178 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.711949 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.713219 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.718721 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.731957 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.747040 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:40.760725 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:40.765622 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.823906 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.826248 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.835462 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.843055 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:40.843746 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.844712 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.851426 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.865021 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.867942 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:40.873070 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.879492 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.881191 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.886076 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.893543 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:40.939774 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:40.969457 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:40.973776 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:40.975479 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:40.979216 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.982496 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:40.995100 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:41.022663 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:41.026239 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.127130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:41.138514 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:41.146982 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.148427 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.150568 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.162405 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.177987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.181954 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.183187 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.188232 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.192838 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.195966 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:41.200526 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:41.223846 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.237237 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.240355 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.242326 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.253132 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.257387 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.268582 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.282554 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.286473 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.291625 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:41.352310 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:41.408387 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.416208 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.416450 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.418008 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.423657 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:41.444254 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:41.452641 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:41.479058 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.479082 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.511807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.518083 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.524524 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.529919 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.537143 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.543839 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.548710 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.548959 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.552778 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.556481 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.561659 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.575064 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.589691 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.594894 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.599483 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.609930 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.627154 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.639089 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.639587 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.646529 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.671425 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:41.716382 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:41.762069 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.779353 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.786007 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:41.796299 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:41.797431 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.798345 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:41.806016 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:41.811452 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.813266 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.847864 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:41.848829 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.859143 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.862248 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.865515 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.873625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.874176 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:41.888213 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.904768 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.917773 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:41.925750 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:41.938225 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.938722 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:41.950006 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:41.967767 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:41.973973 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:41.984746 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:42.009058 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:42.011775 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:42.013058 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.013651 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.019850 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.025554 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.029716 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:42.086808 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:42.119508 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.125202 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.125671 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.135377 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:42.136158 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:42.136915 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:42.137428 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:42.142997 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.151420 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.194684 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.203286 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.204527 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.207079 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:42.214135 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:42.215476 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:42.215880 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:42.218677 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:42.251446 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:42.256611 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.304387 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.307296 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.308398 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.322849 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.325778 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.344061 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.354480 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.406908 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.411801 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.430047 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.430123 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.433637 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.437726 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.443866 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.445432 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.459117 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.464864 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.474143 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.481068 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.483944 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.511463 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.600340 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.603197 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:42.603951 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:42.605536 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:42.614899 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:42.617385 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.618376 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.636034 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.651206 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.666528 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.674551 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:42.682672 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:42.683478 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.685272 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.692035 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.694836 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.703177 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:42.710335 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:42.712716 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.713303 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.715874 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.720774 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.740320 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.742729 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.759516 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.759559 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.761948 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:42.774636 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.787464 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.791949 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.792822 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.803850 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:42.814446 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:42.927874 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.932020 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:42.936756 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:42.944324 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:42.955662 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:42.956051 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:42.966104 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:42.967900 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:42.976449 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.003257 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.004439 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.023299 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.026387 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.028250 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.031214 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.051050 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.058972 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.062252 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.087971 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.095542 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.098393 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.102429 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:43.122038 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:43.155956 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.187256 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:02:43.208143 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.210274 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.215394 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.216795 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.225517 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.235179 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:43.236649 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:43.257447 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.266577 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.274641 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.279500 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.289651 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.295884 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.301154 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.305120 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.368607 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.369612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.372462 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.383826 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.396050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.403250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.407037 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.515283 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.515753 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.524297 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.528849 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.530203 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.533397 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.544557 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.548851 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.550895 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.556006 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.562889 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.563131 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.581001 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.583201 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.588022 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.592211 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:43.603732 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:43.610975 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.621484 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.627204 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.631953 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:43.661856 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:43.669262 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.713522 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.717185 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.723096 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.724030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.728320 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.735050 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.742419 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.808504 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:43.820681 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:43.830782 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.832898 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.835340 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.841871 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.861212 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.861660 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:43.904744 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 08:02:43.905697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:43.908977 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:43.913165 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:43.919061 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:43.927260 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:43.948482 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:43.967540 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.008966 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.013267 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.014853 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.029363 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.048765 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.061901 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.068967 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.077054 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.081892 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.082132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.082376 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.091780 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:44.104676 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:44.128346 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:44.128410 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:44.131659 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.206354 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.208212 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.208436 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.208501 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.216861 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.222630 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:44.224476 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:44.252007 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.284241 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.285290 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.286247 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.290326 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:44.295892 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:44.314143 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:44.326409 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:44.334733 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.361615 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.371858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.390669 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.394979 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.399382 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.410943 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.414220 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.442428 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.446177 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.447356 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.447738 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.450557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.455092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.470350 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.474527 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.527087 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.529875 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.530787 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.534261 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.534895 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.536747 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.553439 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:44.575330 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:44.583902 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:44.617913 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:44.630611 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.631091 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.631651 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.632750 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.636106 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.641443 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.673496 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.720634 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.722151 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.737087 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.739145 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.741577 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.742817 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.746309 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.746457 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:44.748615 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:44.756264 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:44.771926 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.779670 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.791996 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.794485 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.795297 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:44.798498 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:44.834549 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.897660 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.904018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:44.907183 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.912654 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.919410 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.921450 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:44.925539 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:44.932997 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.936381 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:44.943357 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:44.953385 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:44.958957 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.959374 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:44.966113 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:44.970721 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.052956 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.060535 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.068244 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.069010 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:45.085440 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:45.090375 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.090395 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.095516 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:45.152335 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:45.178174 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.181373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.191131 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.195027 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.195255 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.195278 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.197935 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.199708 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.214993 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.215891 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:45.229322 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.232506 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:45.232607 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.240134 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.249569 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:45.274861 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:45.287324 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.366681 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.369882 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.371515 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.373575 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.373673 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.374759 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.375942 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.388677 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.391109 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.394841 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.400029 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.419992 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.428005 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.432227 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.459218 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.464566 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.478360 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.479626 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.491004 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.492130 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.497364 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:45.535280 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:02:45.535491 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:45.622123 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.625125 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.628345 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.631685 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.632040 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.633397 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.640113 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.663616 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.665568 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.688335 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.699725 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.700625 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.701450 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.706950 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:45.710454 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:45.718433 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:45.719473 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.724958 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:45.728653 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:45.744395 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.744638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:45.757299 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.767228 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.775035 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:45.905090 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.905411 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:45.918138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.920351 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:45.957201 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.958093 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:45.959223 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.961850 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:45.962347 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:45.962639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.982779 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:45.993300 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.006295 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.008189 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.010777 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.019040 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.021252 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.021915 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.033895 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.056535 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.057123 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.057701 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.058429 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:46.059338 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:46.061569 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.085795 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.092387 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.103204 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.103229 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.122404 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:46.177150 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:46.193564 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:46.250177 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.261571 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.263397 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.271394 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:46.280625 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:46.286345 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.290947 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.294947 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.310418 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.312176 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.324944 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.327482 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.340518 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.347519 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.352113 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.356410 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.359606 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.362433 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:46.365748 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:46.376367 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.418812 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.420780 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.441778 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.511070 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.513330 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.530575 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.533445 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.548497 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.555013 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:46.558849 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:46.568698 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.579768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.580232 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.593689 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.601567 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.609148 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.609699 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.610018 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.656674 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.657758 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.658752 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.660332 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.673036 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.682020 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.695415 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.745788 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.751605 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.753880 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.756707 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.759769 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.762939 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:46.781394 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:46.792116 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:46.798372 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.857959 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.862603 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:46.863322 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.864945 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.868033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.881727 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.889825 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:46.941401 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:46.954670 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:46.955447 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:46.958197 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:46.963078 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:46.963306 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:46.973392 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.974020 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:46.978343 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:46.978952 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.003445 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.011747 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.029021 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:47.033691 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:47.038067 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.043416 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.079082 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.096258 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.096271 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.115504 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.118786 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.123039 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.127081 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.127863 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:47.190943 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:47.195619 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:47.202075 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:47.206158 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.209486 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.212374 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.212713 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:47.222175 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:47.225268 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.227952 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.239907 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:47.263924 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:47.290035 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.290244 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.298817 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.300747 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.309058 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.342581 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.365306 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.366799 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.373203 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.379107 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.387765 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.389918 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.392796 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:47.401669 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:47.429283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:47.449355 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:47.509297 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.511404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.512355 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.519706 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.519814 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.523638 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:47.563979 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:47.582170 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.605554 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:47.608721 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:47.615783 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.616113 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.622280 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.637144 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.654752 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.663414 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.664304 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.664877 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.665279 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.676471 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:47.682984 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:47.686775 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.686807 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.694232 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.699929 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.712682 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:47.715509 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:47.716583 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.742879 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.745471 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.754101 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.767958 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:47.812167 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:47.814231 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.815075 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.817467 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.825111 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.825430 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:47.826022 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.835137 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.918795 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:47.922230 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:47.924225 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:47.925350 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.925694 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:47.931444 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:47.940204 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:47.990657 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:48.084620 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.086653 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.096215 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.096230 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.105573 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.125900 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:48.127408 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:48.152771 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.167369 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.170625 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.180805 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.181886 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.196453 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:48.200436 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:02:48.208891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.214815 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.234586 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.236106 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.239250 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.241060 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.242772 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.247109 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.256264 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.261575 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.270056 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:48.270463 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:02:48.274123 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.283121 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.291859 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.303949 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.336300 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.402248 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.406868 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.411037 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:48.428018 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:48.436169 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.446976 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.456248 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.457696 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.510331 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.516597 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.518678 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:48.522845 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:48.523273 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.523993 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.532318 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.532969 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.616719 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.618791 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.629493 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.645931 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.647331 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.648779 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.649325 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.650591 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.654461 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.666957 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.672624 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.678665 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.686864 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.697965 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.704295 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.704957 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.729841 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.736429 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.745453 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.747488 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.764087 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:48.789733 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:48.843351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.849449 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.852751 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.856405 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.860893 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.861461 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.883522 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.928225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.931998 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.945561 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:48.945833 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:48.949190 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.949316 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.949426 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.949992 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.964941 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:48.978469 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:48.979696 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:48.981995 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:48.983080 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:48.992601 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:48.997431 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:49.040925 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:49.043466 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:49.114308 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 08:02:49.150444 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.150985 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.156530 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.158805 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.160171 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.171886 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.190796 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.192653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.234411 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.235927 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.242851 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.243363 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.253731 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.258169 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.264580 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.264896 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.267069 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.282558 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.285830 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.286521 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.289021 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.297500 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.313830 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.317453 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.319386 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.341933 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.363438 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.365321 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:49.371243 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:49.485227 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:49.489107 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.498456 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.504918 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.504918 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.510154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.517453 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.530593 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.532905 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:49.546561 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:49.549342 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:49.562283 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.566294 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.575361 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.575579 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.578773 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.583098 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.609997 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.619326 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.622081 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.622279 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.627074 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.640209 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:49.641031 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:49.642005 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:49.679641 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:49.680622 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.683164 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.684511 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.689429 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.689890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.698115 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:49.698439 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:49.722673 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:49.802384 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:49.808980 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.820125 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:49.821506 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:49.822242 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.826520 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.831085 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.839744 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:49.845696 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 08:02:49.870795 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.886638 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.891808 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.893965 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.904281 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:49.907753 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:49.910331 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.912637 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.913961 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.927500 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:49.929651 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:49.936159 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:49.942018 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:49.964787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:49.980857 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:49.982616 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:50.067678 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:50.084720 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.088017 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.088431 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.094017 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.121143 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.122229 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.128511 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.128782 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.130134 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.131871 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.155169 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.162900 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:50.166626 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:50.172170 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.181178 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.258773 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.261373 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.261641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.268216 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.275105 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.295897 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.298077 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:50.310868 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:50.367018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.367842 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.368489 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:50.376821 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:50.384742 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.395231 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.410673 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.416029 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.431506 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.434398 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.445739 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.448576 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.457896 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.461088 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.474834 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.509027 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.512914 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.519858 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.520272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:50.522071 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:50.522989 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.543118 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.552371 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:50.572751 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:02:50.597241 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:50.603713 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.607210 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.613415 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.628151 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.634565 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.642714 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.645557 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.648506 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.656909 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.671933 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.676035 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.681865 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:50.683291 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:50.683495 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.698372 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.826826 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.832068 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.835726 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:50.839821 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:50.840782 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.855419 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.868646 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.884351 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.884792 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.889053 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:50.892226 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.898118 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.899842 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.901372 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.923480 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.941170 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:50.950817 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.951848 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:50.967159 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:50.974748 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:50.984512 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:50.984902 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:50.991372 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:51.005542 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.016281 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.023403 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.025655 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.030459 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.040455 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.060031 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:51.111117 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:51.160666 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.161084 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.171756 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.171936 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.190099 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.193454 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:51.215916 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:02:51.218128 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:51.232249 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.245927 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.251125 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.259513 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.259980 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.269449 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.300477 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.303862 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.312103 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.314323 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.320216 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.335145 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.336246 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.362459 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.366088 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.386775 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.387001 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.391180 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.393877 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.397771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:51.401227 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:51.403808 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:51.404342 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.419440 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.514748 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.515841 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.520771 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.524756 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.535737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.555269 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.562631 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.577471 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.578871 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.587205 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.598670 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.604991 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.605026 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.624919 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.637264 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.641813 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.650241 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.650897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.655254 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.656834 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:51.673611 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:51.673766 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:51.688445 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:51.709506 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:51.757383 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.761907 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.762300 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.787737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.789523 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.797467 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.800801 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.811152 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.811226 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.832491 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:51.834745 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.837903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.857837 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.860919 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:51.869098 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:51.936905 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:51.946574 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:51.949162 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:51.954948 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:51.972122 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:51.987648 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:51.997660 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:51.998414 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.013921 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.016306 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.018131 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.023080 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.028220 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.028914 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.053829 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.127959 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:52.139147 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:52.139693 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:52.143013 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.149128 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.150240 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.151434 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.152482 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.153285 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.157613 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.160017 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:52.161991 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:52.162609 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.176069 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.183967 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.227618 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.238802 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.239085 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.242797 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.263197 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.279366 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:52.286995 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:52.287875 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.297432 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.298413 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.374394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.375524 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.383236 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:52.384562 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:52.385636 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.393824 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.402068 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.420567 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:52.466403 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:52.466796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.474747 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:52.479173 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:52.481398 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.482219 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.487721 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:52.487931 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:52.495469 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.497215 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.502030 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.503850 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.531029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:52.542780 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:52.544600 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.553511 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.556422 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.557976 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:52.649022 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:52.649739 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:52.656922 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.663702 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.668125 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.675630 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.684406 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.686332 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.688935 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.710765 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.720525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.721489 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.722581 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.728582 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:52.743940 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.746634 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:52.747864 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.841025 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:52.850361 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:52.860242 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.863065 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.865434 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.876005 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:52.877642 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:52.878799 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.882790 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:52.891126 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:52.893383 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:52.895860 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.908535 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.917912 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.931195 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:52.948679 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:52.993589 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:53.003917 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:53.004947 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.005220 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.008049 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.012776 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.021005 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.024909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.097625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.101065 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.106171 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.116336 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.119113 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.119131 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.126466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.135836 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.141610 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.144209 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.146944 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.152010 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.160703 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.170513 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.262094 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.265172 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.273674 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.277575 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.295598 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.296525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:53.303570 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:02:53.311412 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.367343 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.368894 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.376382 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.398332 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.401417 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.410540 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:53.412278 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:53.440675 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.458293 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:53.459421 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:53.460901 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.465646 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.469221 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.479660 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.496722 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.529613 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.547140 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.550194 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.550891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.561918 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.603922 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.611896 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.620024 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.621858 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.637706 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.649091 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.662150 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.668241 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.681248 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.684178 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.701553 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:53.703896 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:53.704313 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.704372 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.706552 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.729941 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.742656 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.749840 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.772449 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:53.779300 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:53.781152 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.784288 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.796875 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.801724 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.806072 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.830325 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:53.876569 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:53.912758 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:53.920114 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.922106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:53.932152 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:53.946190 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.947716 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.951543 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:53.953555 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:53.959375 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 08:02:53.976877 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:53.977386 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:53.996559 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.000393 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.006896 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.017657 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:54.028198 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:54.049199 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.081640 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.093178 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.095711 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.099277 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.101796 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.117820 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.121615 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.124563 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:54.142012 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.148542 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.157716 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.166577 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.169358 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.172277 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.193080 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.272336 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.272919 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.276741 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.282734 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.289289 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.297914 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.310243 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.317023 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.321345 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.342523 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.343866 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.347007 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.349951 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.363593 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:54.378304 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:54.435207 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.435743 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.438660 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.447405 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.459829 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.461597 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.486703 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.524868 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.530761 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.533049 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.548362 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.567435 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.583257 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.590605 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:54.609406 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.609453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:54.615631 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.621877 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.623960 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.626216 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.633069 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:54.646142 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:54.655067 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.655205 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.665817 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.707026 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:54.709652 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.716650 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.719522 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.720443 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.721367 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.740139 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.743932 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.746507 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.755628 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.760766 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.762322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.799962 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.804353 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.812677 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.884810 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.889240 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.893735 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:54.894326 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:54.913451 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:54.920156 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.942137 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:54.947934 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:54.952806 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:54.964357 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:54.968230 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:54.976063 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:54.994321 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:54.999981 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.003388 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.012890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:55.020228 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 08:02:55.033293 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.039878 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.040163 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.050544 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.068085 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.071858 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.077852 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.146144 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.148437 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.149950 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.157094 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.165586 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.178290 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:55.178756 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:55.180188 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:55.232949 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:55.252980 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.260092 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.261936 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.264202 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.266635 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.275380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.295167 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.311506 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.312678 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.317847 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.319519 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.330061 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.330091 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.332154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:55.341719 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:55.343850 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.377910 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.418236 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:55.479295 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.479597 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.484290 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.485539 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.504724 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.505352 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.517753 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.522409 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.526453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.526870 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.530025 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.543481 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.555241 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.557236 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:55.623519 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:02:55.624215 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:55.625547 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.628252 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.631302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.640961 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.642515 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.648622 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.658699 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.710672 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.712505 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.713523 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.736114 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.740397 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.767454 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:55.770112 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:55.770193 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.829165 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.834867 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.843133 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.848187 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.849555 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.856488 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:55.893076 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:55.916256 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:55.963896 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:55.972088 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.974350 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:55.974478 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.979393 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:55.984771 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:55.996713 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:55.997503 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.009016 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.014119 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.019093 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.019621 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.020551 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.025806 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.042083 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:56.043303 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:56.048302 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.048825 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.061995 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.072378 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:56.074780 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:56.075285 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.080601 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.208652 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.210345 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.213510 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.219053 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.221699 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.244033 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.259768 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:56.282184 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:56.316898 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.317518 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.319260 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.326875 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:56.333721 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:02:56.334326 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:56.335620 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:02:56.336849 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.356879 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.375603 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.383070 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.383564 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.398855 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.406560 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.409674 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.418704 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:56.420225 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:56.448987 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.449357 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.455135 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.460588 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.464649 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.465224 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.466710 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:56.487460 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:56.492733 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:56.568954 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.569707 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.569777 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.574307 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.585186 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.588843 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.610633 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.661162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:56.673363 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:56.675091 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.676775 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.686594 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.691704 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.696774 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.709956 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.731071 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.732384 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.738013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.749557 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.760288 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.807067 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.813406 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:56.816439 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:56.832325 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.834688 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.844034 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.849751 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:56.852416 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:56.861083 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.862048 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.863755 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.872009 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.875072 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:56.893218 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.893788 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.926880 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.928437 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.947516 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.978333 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:56.979612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:56.980842 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:56.984743 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:56.986454 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:56.999759 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:57.046144 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:57.049866 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:57.054144 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:57.069236 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.070482 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.070868 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.079003 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.084261 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.098723 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.102241 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.217847 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.219614 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.220477 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.227203 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.232734 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.233872 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.256214 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.256554 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.258339 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.289099 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.289433 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.290341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.302619 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:57.325309 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:57.334565 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.369443 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.369702 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:57.382683 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:57.385623 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.392354 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.392545 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.403530 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:57.412004 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:57.422739 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.475369 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.476979 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.494671 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:57.499100 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:57.501353 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.505503 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.510746 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:57.517163 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:57.533680 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.591590 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.598640 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.601799 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.615515 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.619678 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.625269 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:57.625820 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:57.630453 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.631610 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.644425 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.658581 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.658679 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.659441 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.662380 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.672291 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.694566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:57.695442 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:57.700379 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.710595 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.724299 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.725265 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.726104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:57.744735 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:57.766466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:57.779685 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:02:57.783532 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.807060 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:57.833520 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.840936 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.844830 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.849323 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.854377 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.857378 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.869127 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.873862 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:57.877563 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:57.878283 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:57.881084 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:57.883874 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:57.910593 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:57.928102 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:57.997997 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:58.066031 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.068456 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.074960 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.076012 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.092017 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:02:58.093924 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:58.094156 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.097023 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.109838 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.112056 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.112473 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.120118 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.122150 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.133949 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.160492 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.173225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.177006 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.179147 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.198508 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.199550 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.203919 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.204177 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.273067 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.273910 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.276062 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.279474 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.281272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.285894 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.326890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:58.359956 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 08:02:58.364584 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:58.399277 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.400245 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.408550 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.420411 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.423477 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.437934 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.439739 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.490723 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.497699 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.499488 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.499724 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.499934 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.503949 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.505354 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.544688 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.550279 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.564558 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.565459 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.578730 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.597400 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.599205 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:58.630367 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:58.655910 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.656431 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.663125 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.663174 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.679473 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.707115 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.712160 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.712522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.720132 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.723501 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.744421 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.754125 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.761523 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.764386 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.777050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.780824 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.781939 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:58.791445 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:58.804569 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.807402 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:58.807696 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.818267 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:58.850404 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.876152 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:58.925098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.930977 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:58.941547 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:58.941812 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.947060 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.948009 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:58.955109 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:58.972193 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.979575 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:58.989588 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.003968 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.005692 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:02:59.011039 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:02:59.023094 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:59.024705 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:59.026458 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.034966 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.051414 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.064967 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.069548 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.083447 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.094779 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.095688 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:59.147608 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:59.152349 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:59.192736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.192987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.193765 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.194943 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.198104 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.198724 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.200166 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:59.208711 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 08:02:59.255064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.256870 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.260056 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.271224 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.285128 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.287796 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.300258 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.360078 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.378345 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.384517 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:02:59.387804 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:59.396629 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.408922 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.410976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.412065 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:02:59.415099 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:59.431628 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.432341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.435994 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.446301 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.469417 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.470250 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.504644 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.505897 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.509267 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.517470 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.531759 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.534683 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.535297 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.535586 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:59.564939 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:02:59.623781 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.628839 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.633325 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.645599 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.648289 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.668848 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.671597 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.703882 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.708606 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.717060 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.718278 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.742197 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:02:59.744143 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:59.748466 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.749892 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.750159 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.763597 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.767135 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.784921 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.790946 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.801620 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:02:59.806578 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:02:59.815503 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.909626 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:02:59.915848 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.927284 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.927435 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.931015 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:02:59.945426 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.959015 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:02:59.986146 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:02:59.989193 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:02:59.994782 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:02:59.997152 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:00.000434 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:00.004949 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.007083 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:00.010533 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:03:00.016767 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.021449 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.024030 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.024603 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.024844 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.037377 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.083503 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.104629 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.110834 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.113822 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:00.116209 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 08:03:00.135353 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.137064 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.140563 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.142501 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:00.145159 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:00.179546 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:00.224771 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.233592 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:00.275312 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.278774 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:00.280073 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:00.287979 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.289129 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.297552 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.328316 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.334298 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.338328 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.345853 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.353267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.354522 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.371024 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.376952 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.381309 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.388289 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.402630 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.404088 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.405184 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.410617 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.422856 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.443823 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:00.461012 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:00.474587 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.484452 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.484724 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.488749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:00.499749 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:00.500499 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.515336 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.520692 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.592011 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.597121 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.597646 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.601719 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.602104 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.627146 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.637969 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.661046 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.662027 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.667254 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.668653 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.671725 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.681874 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.708884 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:00.765627 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:03:00.766863 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:00.810815 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.812395 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.816444 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.817021 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.842298 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.843421 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.857064 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:00.858752 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.862318 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.862325 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.881582 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.883361 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.886808 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.891222 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.945035 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:00.948002 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.949124 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.949657 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:00.954389 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:00.961575 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.962846 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.987025 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:00.987706 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:00.997537 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.000756 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.023510 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.024187 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:01.037084 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:01.045049 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.150414 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.154491 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.161886 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.169217 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.172816 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.206087 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:01.206326 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:01.207326 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:01.208498 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:01.233633 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.253352 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.256242 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.266580 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:01.275846 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:01.277886 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.280823 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.282115 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.283438 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.285813 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:01.334895 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:03:01.338444 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:01.342630 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:01.345577 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.348469 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.349973 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.354959 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.367967 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.387609 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.400370 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.402586 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.412864 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.426146 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.429716 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.431022 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.438625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.448261 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.449095 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.464579 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.476500 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:01.483546 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:01.489660 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.490221 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:01.508972 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.528694 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:03:01.567518 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.569852 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.572674 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.598184 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.598345 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.598577 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.598695 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.633425 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.636318 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.636714 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.643177 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.645409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.653092 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.661901 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:01.729569 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:01.740265 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:01.762588 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.763602 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.766799 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.777201 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.783788 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.799243 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.802446 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.855924 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.858067 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.859629 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.863015 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.873301 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.899329 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.902966 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.908321 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:01.914263 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:01.925379 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:01.929574 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:01.936988 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.938760 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:01.942404 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.988420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:01.994725 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:01.997030 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:01.998658 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:02.002625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.005383 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.006513 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.009959 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.011405 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.064713 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.073817 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.088088 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.095727 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.097134 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.102739 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.113255 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:02.114043 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:02.223315 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:02.225726 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:02.230443 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.233357 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.239861 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.240531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.242558 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.254016 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.257117 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.272284 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.284387 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.296069 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.310520 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.316548 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.333178 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.339075 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.350096 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:02.356550 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:02.357681 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.363668 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.371389 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.375075 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.381714 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:02.389770 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:02.467819 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.477234 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.477329 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.486460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.497113 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.507560 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:02.508314 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:02.523486 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.545332 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:02.548272 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:02.554160 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.554692 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.556396 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.562215 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.569813 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:02.582612 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.582696 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:02.590309 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:02.605497 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:02.623271 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.625008 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.650377 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.655836 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.656571 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.657431 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.662341 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.663426 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.675879 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.688127 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.710611 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.713011 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.723078 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.724592 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.816752 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.818830 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.819662 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.836050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.849409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.851682 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.854988 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.860567 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.863955 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.864584 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.867326 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.867467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:02.883409 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:02.883953 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:02.902197 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:02.974224 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:02.978962 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:02.983033 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:02.985497 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:02.987285 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:02.998386 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:02.998584 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.011691 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.020430 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.020586 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.020718 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.021548 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.025615 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.030730 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.037850 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:03.061363 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:03.142004 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.146037 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.146871 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.147794 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.148530 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.149551 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.201619 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.204167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.213119 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.218741 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.220405 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.226034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.227387 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.244465 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:03.263549 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:03.304605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.307902 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.313459 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.333681 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.339231 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.342253 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:03.343406 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:03.356534 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:03.368026 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:03.426214 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.433842 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.434782 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.436694 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.450721 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.457829 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.470149 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:03.505254 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:03.519285 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.523571 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:03.529109 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:03.533046 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.536461 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.537161 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.545325 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.547166 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:03.556917 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:03.574110 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.593392 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.596634 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.608124 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.611327 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.622401 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.643698 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.654997 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.658337 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.661373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.671321 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.671939 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.677701 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.697139 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.757999 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.764220 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.766362 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.768605 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.787050 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.791855 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.806341 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.848999 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:03.854775 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.855613 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.856627 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.864018 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:03.868683 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.870146 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:03.873327 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.886270 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:03.887918 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 08:03:03.892391 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:03.915675 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:03.916374 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.923194 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:03.937119 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:03.952328 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.970851 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:04.007333 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:04.049851 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:04.055330 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:04.076102 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.077742 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.085595 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.090606 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.097445 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.098821 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.101634 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.108566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.118602 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.122056 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.130807 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.131780 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.133150 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.136065 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.139752 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.152428 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.164976 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.172297 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.190868 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.206976 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.210635 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.214461 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.236378 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.241575 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.248355 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.254056 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.260924 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.272531 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:04.279837 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:04.354768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.354876 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.359117 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.366076 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.375918 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.379583 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.379976 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:04.426054 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:04.464375 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.466105 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.478511 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.489075 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.489702 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.503013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.507380 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.519553 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.546583 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.547586 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.547856 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.551118 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.559248 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:04.568378 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:04.571712 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.573453 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.585521 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.596318 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.599179 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.601667 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.605806 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.611392 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:04.658710 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:04.696934 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.698330 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.704718 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.709836 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:04.711102 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:04.715338 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.743647 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.755676 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.784289 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.785876 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.792171 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.799243 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:04.803153 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:04.811133 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.828752 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.846757 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.848948 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.858834 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.860443 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.871069 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.880142 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.900027 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.900626 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:04.919929 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:04.926534 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:04.928306 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:04.933144 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:04.940404 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:04.961457 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:04.977037 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.986325 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:05.002704 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.101655 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.102537 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.105999 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.106695 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.107263 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.108953 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.118337 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.124176 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.130036 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.134778 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.146909 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.151077 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.153240 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:05.166977 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:05.171252 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:05.196740 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:05.215673 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.240227 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.249312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.258893 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.259510 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.272868 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.274248 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:05.280817 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:05.299147 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.307902 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.313712 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.322822 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.327602 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.343154 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.347777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.358322 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:05.396356 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:05.440375 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.444365 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.463151 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.471007 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.471436 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.471755 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.473599 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.474390 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.481384 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.497248 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.500451 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.508059 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.517863 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:05.531571 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:05.532229 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:05.590991 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:05.592213 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.600644 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.602751 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.603364 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.603714 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.604384 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.632625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.633924 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.652462 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.663100 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.663353 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.669775 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.681613 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.692489 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.739204 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.740595 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.741238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:05.741673 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:05.749406 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.750417 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.751669 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.767925 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:05.774961 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:03:05.776353 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:05.846746 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.851213 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.853229 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.857306 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.857934 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.871179 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:05.881520 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.924679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:05.927805 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:05.931653 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.949437 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:05.953176 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:05.959071 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:05.983290 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:05.991937 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:06.051535 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.055136 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.057140 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.077106 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.081418 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.086205 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:06.094303 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:06.101186 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.147064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.157140 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.159672 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.159864 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.175005 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.185298 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:06.189101 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:06.192945 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.199162 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.212733 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.223938 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:06.226842 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:06.228454 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.229293 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.230221 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.235070 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.236910 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.253082 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.256345 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.258205 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.261166 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.263715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.309979 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:06.330076 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:03:06.409390 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.409912 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.424548 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.429177 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.434227 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.441463 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.442288 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.477627 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.477754 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.484953 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.501148 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.509730 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.512262 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.515118 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:06.519869 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:03:06.520524 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:06.534013 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.537217 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.541342 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.547569 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.547847 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.557222 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.567966 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:06.591482 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:06.610024 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:03:06.665236 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:06.683198 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.694267 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.694737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.724406 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.731285 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.736761 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:06.741024 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:06.747689 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.752256 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.755780 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.759776 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.761767 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.780058 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.781282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.796686 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:06.797047 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:06.810128 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.834791 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:06.835417 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.836155 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.840519 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:06.846196 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.856353 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:06.856729 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.871935 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.873162 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:06.874361 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.875291 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:06.880638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:06.884514 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:06.953317 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:07.012634 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.018597 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:07.022677 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:07.032545 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.037801 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.041628 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.044617 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.079109 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.119087 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.135754 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.144300 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.145107 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.151435 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.177978 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.180771 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.181025 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.181680 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.182934 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:07.191732 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:07.205081 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.208813 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.217885 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.218627 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.225275 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.226166 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.237974 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.244050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.248827 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.261926 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.270192 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.355835 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:07.360152 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:07.362649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.369357 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:07.375460 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:07.386138 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.389343 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.395920 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.410387 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:07.464893 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:07.467701 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.470361 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.475299 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.483778 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.489028 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.506104 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.520679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.522908 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.524566 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:07.534025 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:07.549928 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:07.553849 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:07.556979 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.557549 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.571805 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.574331 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.575915 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.580725 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.588187 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.592279 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.593719 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:07.610745 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:07.623232 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.627261 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:07.708843 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:07.725062 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.735844 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.753245 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.754422 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.757261 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.766284 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.783077 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.787024 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.809878 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.812916 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.818979 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.819205 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.824137 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.851730 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.852537 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.853564 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.866366 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.874341 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.880514 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:07.881275 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.881938 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:07.945998 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:07.950773 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:07.957434 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:07.959092 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:07.963156 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:07.966799 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:07.971879 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:07.975672 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:07.986366 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.005517 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.015096 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.015188 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.044248 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.046077 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.054527 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.056098 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:08.103984 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:08.133885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.141913 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.148239 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.148648 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.150498 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.166800 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.185021 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:08.195029 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:08.237898 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.239022 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.240448 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.261682 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.272207 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.272524 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.277751 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.356252 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.365463 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.366403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.368245 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.375170 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.375849 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.377094 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:08.399084 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:08.414866 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:03:08.425401 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:08.430749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:08.443518 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:08.449290 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.449477 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.457517 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.459657 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.465422 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.473924 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.474622 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.480173 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.480782 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.507287 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.516239 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.518023 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.544303 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:08.553104 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.561216 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:08.628609 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.630642 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.631110 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.641201 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.642851 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.651125 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.662895 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.666098 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.674770 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.677018 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.678160 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.678525 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.679452 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.719266 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:08.737092 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:08.775341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.784476 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:08.795513 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:08.797032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.799647 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.802783 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.811000 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.817680 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.821838 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.833739 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.836004 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.842873 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.857543 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.874667 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.877323 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.937955 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:08.939785 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:08.943136 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:08.944494 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:08.953045 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:08.960031 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:08.972781 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:08.990603 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:08.992172 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.006134 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.015825 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.018095 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.019163 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.024679 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.041671 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.123991 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.128808 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.130599 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.135762 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.146001 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.175500 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.178160 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.208364 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:09.209980 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:09.216571 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.220191 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.222734 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.222745 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.236305 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.240003 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.291548 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.296820 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.313212 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.314407 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.318696 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.323078 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.342638 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:09.363159 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:03:09.364367 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:09.384003 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.391896 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.396739 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.397467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.407631 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.408609 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.409399 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:09.418002 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:09.476903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.477428 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.487455 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.493620 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.494136 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:09.501876 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:09.504096 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.515632 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:09.565829 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:09.579845 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.582842 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.603713 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.604413 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.613437 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.620554 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.636633 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.651633 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.657477 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.661193 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.672132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.675227 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.676530 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.705693 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.724686 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.737272 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.739109 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.752197 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.757349 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.760609 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.767225 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.770077 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.784994 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.787135 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.787922 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.789880 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:09.802186 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:09.803028 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.808660 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.879020 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.889387 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.890225 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.892803 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:09.893189 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.894112 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:09.924762 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:09.931399 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:09.971964 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:09.975917 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:09.980543 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:09.986349 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:09.989131 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:09.996012 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:09.997292 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.022197 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:10.038949 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:10.068406 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.081759 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.093117 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.094815 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.095774 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.096434 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:10.103338 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:10.121440 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.191989 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.193378 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.197718 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:10.203639 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:10.225628 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.226316 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:10.231005 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:10.235982 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.239485 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.242210 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.248356 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.248943 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.255949 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.260087 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.261655 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:10.270821 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:10.285266 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:10.315879 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.316196 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.329777 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.334288 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.344365 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.351135 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.380553 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.395217 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.442715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.443634 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.449447 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.450194 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.450209 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.450937 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.450938 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:10.468789 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:10.541128 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.549547 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.559892 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.561401 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.563016 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.565868 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.568212 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.570463 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.576005 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.581532 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.606716 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:10.610177 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:10.613637 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.619705 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.627370 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:10.661332 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.700897 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:10.706619 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.707031 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.714006 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.715118 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.728821 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.729256 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.756081 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.757941 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.759310 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.762518 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.773162 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.774341 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.790393 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.805460 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.845516 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:10.850911 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.854631 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:10.854975 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:10.857453 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.864930 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.887766 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:10.938442 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:03:10.939878 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:10.972764 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:10.986186 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:10.988866 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:10.992908 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.004911 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:11.014333 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:11.021481 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.027057 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.040597 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.052024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.068017 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.068773 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.076923 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.077630 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.080415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.081322 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.095307 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.107892 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.109513 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.119179 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.120590 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.132692 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:11.158315 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:11.245397 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.245551 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.248028 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.259892 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.286998 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.291482 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.324045 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.328079 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.330459 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.334676 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.344941 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.348424 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.366293 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:11.380939 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:11.397951 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:11.401353 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:03:11.417361 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.421463 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.434435 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.463446 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.472338 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.472925 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.475328 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.499105 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.502627 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.510885 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.522831 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.527148 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.547039 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.556805 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.617033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.627418 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:11.636247 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:11.642558 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.644655 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.646023 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:11.650733 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:11.651546 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:11.668344 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.671680 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.674672 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.675673 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.680018 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:11.680743 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:11.685014 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.689071 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.711318 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.713971 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.722316 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.733322 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.750543 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.752321 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.752509 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.753629 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.790445 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.810461 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.817103 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.826057 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:11.828058 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.828270 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.855022 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.860114 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:11.868775 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:11.944626 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:11.951815 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:11.960413 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.965417 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:11.977739 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:11.978286 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:11.978590 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:11.980816 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:11.994525 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.000489 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.013664 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.023183 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.023617 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.029719 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.036850 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:12.082218 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:03:12.098627 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:12.100170 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:12.110399 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.115971 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.118284 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.131645 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.132104 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.134562 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:12.153591 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:12.210680 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.217067 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.220182 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.232445 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.242353 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.247361 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.249379 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.251855 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.259392 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.261993 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.267335 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.268913 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.275651 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:12.276840 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:12.297223 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:12.306589 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.409550 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.411278 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.414525 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.423038 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.442044 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.446153 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:12.452988 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:12.461553 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.482349 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.493742 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.500497 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.503638 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.504021 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.505070 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.511787 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:12.515756 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:12.552624 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:12.561858 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:12.561942 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:12.570034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.572402 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.573051 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.574303 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:12.588263 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:12.589700 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.589866 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.618130 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.618415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.620126 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.622659 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.627877 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.665944 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.667799 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.728730 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.735076 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.743934 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.754886 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.756834 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.762777 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.763204 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.837469 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:12.851467 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.853289 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:12.870790 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:12.876052 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.879884 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.880728 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.887012 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.889696 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.898826 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:12.903100 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:12.904645 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.905881 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.918695 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.920869 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:12.932708 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:12.932968 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.934013 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.937562 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:12.953173 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:12.956198 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:12.959524 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:12.976246 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:12.976963 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:13.100423 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:13.126552 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.129895 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.132333 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.134995 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.147996 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.148360 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.152753 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.170661 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.173527 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.181537 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:13.181967 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:13.215353 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.220820 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.223903 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.225822 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.232743 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.233269 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.260635 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.274267 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.307689 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.312260 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:13.314752 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:13.318001 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.320107 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.325183 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.331305 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.338141 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.351711 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.360857 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:13.361934 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:13.389254 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.438331 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.439812 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.443885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:13.449358 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:13.451272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.458223 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:13.463985 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:13.466444 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.476397 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.510542 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.512537 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.518803 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.520498 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.522862 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.537132 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.547560 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:13.568500 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:13.590341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.592130 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.600186 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.604763 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.608289 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.609958 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.625444 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.693681 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.694133 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.699824 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.700590 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.708130 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.711112 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:13.723556 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:13.726251 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.784833 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.799746 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.803393 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.808094 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.812727 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.817514 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.826385 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.828032 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.835660 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.845841 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:13.870543 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.880538 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.882134 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:13.886179 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.965895 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:13.966356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:13.967777 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:13.978097 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:13.984620 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.026433 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.027101 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.039376 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 08:03:14.051409 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.051672 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.057466 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.063650 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:14.064708 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:14.076303 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:14.080262 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.081394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.090888 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.098212 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.101953 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.105499 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:14.113785 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:14.121387 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.122582 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.126519 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.133889 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.186471 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:14.203976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.209029 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.213678 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.228247 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.231505 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.232161 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.240350 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.321022 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.321274 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.342006 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.347043 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.347537 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.348071 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.361764 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.365155 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.375802 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.379988 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.407662 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.408347 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.420118 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.420418 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.473827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.478302 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.480571 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.502341 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.503436 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.504878 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.512768 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.528184 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:14.529142 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:14.574167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.575443 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.585686 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.585922 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.594558 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.607825 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.608475 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.619652 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.626137 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.631186 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.645426 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:14.647594 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.647912 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.649852 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.660055 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.670140 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:14.738942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.743033 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.750142 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.752508 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.764622 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.767382 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.789361 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.834044 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.836758 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:14.844552 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:14.844658 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.849941 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.850590 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.882551 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.883025 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:14.893731 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:14.898957 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.906167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.920600 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.928771 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.931011 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:14.937885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:14.938108 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.952594 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.952615 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:14.965965 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:14.980144 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.986299 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:14.990909 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:14.992466 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:15.019971 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.066864 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:15.079723 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:15.099176 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.106377 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.123253 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.125241 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.127699 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.141429 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.158994 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:15.161769 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:15.174612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.198668 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.211655 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.220299 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.221047 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.227036 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.229787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.231312 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.231539 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:15.231627 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:15.263048 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.264842 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.273485 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:15.277904 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:15.296011 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.298192 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.308077 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.309115 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.316011 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.316619 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.330514 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.334990 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.339874 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.452253 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.458465 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.460686 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.460719 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.471992 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.483691 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.494861 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:15.519546 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:15.542871 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.544714 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.546541 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:15.558604 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:15.560146 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.560786 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.561692 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.567782 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.583074 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:15.588639 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:15.589398 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.590267 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.605831 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.606978 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.630445 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.630896 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:15.644057 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:15.706949 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:15.747416 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.753127 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.762750 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.768560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.771027 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.782048 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.787510 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.787600 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.790632 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.799397 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.802234 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:15.805950 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:15.819336 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.819917 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.847180 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.854612 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.863118 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:15.863966 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.864511 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.869159 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.870337 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.912757 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.929862 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:15.933444 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.941460 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:15.944737 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:15.945755 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:15.959015 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:15.962589 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:16.049230 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:03:16.049945 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:16.063436 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:16.080441 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.084739 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.091476 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.092434 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.104615 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.105296 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.115269 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.149953 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.168588 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.169878 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.174980 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.175207 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.175622 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.183576 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.185065 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.190186 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.192610 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.207581 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.209565 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.213884 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.244529 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.367192 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.368793 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.372337 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.379423 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.381413 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.386253 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.388906 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.390352 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.393362 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.396510 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.396998 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.407925 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.426697 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.433099 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:16.495421 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.543848 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:16.569087 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.580823 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.582000 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.596594 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:16.606290 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:16.609292 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.620234 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.626210 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.633265 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.636909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.646843 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:16.659387 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 08:03:16.660909 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.660935 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.668901 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.670233 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.674803 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.678566 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.679572 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.701371 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.706269 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.708537 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.715723 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.716343 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.721578 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.727818 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.732451 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.747895 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.756243 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.765261 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:16.814051 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.827073 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:16.827891 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:16.863596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.872874 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.877116 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.887725 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.895977 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:16.897382 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.907466 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.936298 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:16.936628 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:16.937000 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:16.939929 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:03:16.955256 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:16.955513 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:16.958932 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:16.987536 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.054792 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.056349 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.058816 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.064888 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.067407 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.068012 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:17.072979 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:17.086426 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:17.090621 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:17.107828 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:03:17.108547 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.121115 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.127207 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.132516 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.135627 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.137018 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.159404 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.226413 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.230458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.232611 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.233465 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.243935 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.257247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.258072 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.261848 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.263664 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.264026 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.304497 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.304769 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.312075 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.315577 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.433403 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.442557 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.443492 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.451210 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.452153 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.470579 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.475551 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:17.479763 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:17.481716 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.484934 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.486200 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.486675 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:17.498948 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:17.503110 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.507867 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:17.513604 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:17.550899 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.551235 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.553522 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.555795 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.559438 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:17.562304 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:17.566500 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.593577 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.611503 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:17.620437 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.661140 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:17.669238 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.673681 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.681725 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.688761 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.692183 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.710461 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.719120 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.750018 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.754390 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.758570 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.763305 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.766266 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.769167 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.782830 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:17.787684 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:17.849879 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.854529 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:17.854732 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:17.858583 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.865221 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.870556 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:17.887367 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:17.889935 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.915827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:17.928283 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:17.930120 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.932272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:17.934365 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:17.941006 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:17.949105 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:17.950747 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:17.990514 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:18.013957 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:18.020684 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.028035 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.030242 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.036849 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:18.040050 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:18.054651 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.060269 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.069933 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.107089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.127393 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.131755 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.136965 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.142652 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:18.151496 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:18.168988 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.187814 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:18.189765 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:18.191232 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.193440 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.194231 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.209965 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.211069 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.213263 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.214618 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.231477 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.244436 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.246100 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.258714 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.260218 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.286875 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.287257 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.386960 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.386984 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.394512 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.402904 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.406224 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.409694 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.437043 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:18.443807 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:18.464460 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:03:18.475028 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.475950 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.482722 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.502844 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.504652 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.509195 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.515789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.518880 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.536850 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.555073 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.560773 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.567066 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.568710 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.587424 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:18.649609 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:18.663409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.671510 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.674445 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.676954 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.677778 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.690383 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.701447 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.706130 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.716677 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.717445 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.723507 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.732091 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.742194 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.742766 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.800361 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.810483 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.815120 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:18.816370 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:18.817557 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.841107 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:18.843793 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.845032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.916932 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:18.918740 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:18.929375 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:18.934235 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.958714 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:18.959498 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:18.962754 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:18.998443 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:19.004336 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.021581 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.022243 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.022892 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.027565 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.032985 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.058326 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.064918 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.068192 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.073277 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.074587 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.081169 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.086364 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.086613 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:19.092522 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:19.151993 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 08:03:19.198029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.202625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.203639 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.205085 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.205276 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.205276 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.205472 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:19.240260 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:19.254849 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:19.284046 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.288290 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.307658 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.311460 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.313753 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.326845 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.327536 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.328878 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.339188 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.352109 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.363899 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.375890 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.382301 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.383600 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.392823 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.399286 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.405222 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.406338 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.419070 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.432183 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.439018 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:19.460222 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:19.524570 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:19.580091 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.585772 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.601327 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.603638 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.624873 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.641716 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.647183 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:19.654954 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:19.658084 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.658126 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.674289 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.677207 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.679394 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.693123 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.702793 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.709350 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:19.712942 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:19.718228 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.718765 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.722693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.733096 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.735883 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.737210 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.744468 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.749080 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.755843 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.761281 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.774194 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.776103 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.779083 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:19.782131 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.788806 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.816104 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.824440 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:19.826675 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.843458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.855614 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:19.930210 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:19.974440 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:19.976197 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:19.981117 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:19.987603 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:19.990097 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:19.994406 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.009042 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.013898 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.019603 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.027281 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:20.030856 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:20.031072 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.031141 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.058849 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.064045 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.075510 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:20.096743 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:20.109884 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:20.110347 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:20.227162 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.241863 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.243384 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.243695 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.245448 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.245872 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.270576 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.273484 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.274020 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.279538 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.279575 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.282271 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.287117 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.289649 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.294018 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.299729 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:20.302349 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.307163 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:20.327400 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.333165 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.336741 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.378659 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.419360 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.426004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.433912 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.434724 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.437999 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.438761 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:20.442320 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:20.445436 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:20.463626 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.464821 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:20.473300 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:20.473834 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:20.477826 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:20.480885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.499166 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.501209 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.505744 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.517979 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.535104 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:20.566382 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:20.627566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.628018 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.636946 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.637952 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.644316 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.645205 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.648059 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.665481 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.667352 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.667378 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.671154 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.679880 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.686204 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.722044 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.754418 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.760826 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.769157 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.771278 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.775458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.786967 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.787814 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.854045 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.858249 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.861480 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.863600 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.868628 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:20.881302 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:20.891822 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.893479 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.931113 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:20.932459 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:20.934648 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:20.935629 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:20.940207 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:20.946913 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:20.953334 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:21.014361 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:21.026357 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:21.071750 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.073312 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.075351 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.075989 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.077823 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.080852 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.099068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.110280 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.110903 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.112013 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.130388 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.135478 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.156997 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.183835 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.235100 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.241154 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.243111 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.246533 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.258280 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.275068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.276952 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.325266 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.329814 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.335247 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:21.356705 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 08:03:21.363222 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.367682 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.367703 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.386687 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.447935 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.448552 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:21.456298 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:21.463054 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.477943 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.481591 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.481595 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.496528 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.508248 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.516278 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.521967 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.528537 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.534349 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.535070 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.544121 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.558280 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.568087 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:21.580363 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:21.580432 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:21.580407 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.580587 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.582732 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.596339 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.604254 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.655295 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.662909 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.668792 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.669924 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.679803 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.681214 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.698714 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:21.725790 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.726531 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 08:03:21.789752 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.790861 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.793004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.798276 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.803349 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.817409 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.818544 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.819392 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:21.831642 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.832724 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.833168 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.841482 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.859603 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.872143 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:21.872676 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.873742 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:21.877589 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:21.881158 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.892632 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:21.911504 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:21.913347 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:21.970743 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:21.980602 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:21.982056 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:22.026369 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:22.029670 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.032709 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:22.037859 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:22.042425 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.046304 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.057507 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.060251 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.084017 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.123205 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.123466 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.126202 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.131989 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.139389 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.149804 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.166383 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.187176 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.187693 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.192440 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.198247 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.198802 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.200516 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:22.211797 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:22.230631 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.258335 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.263284 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.263530 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.268683 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.277042 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.285753 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.286901 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:22.416613 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:22.417702 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.444412 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:22.448525 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:22.450073 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.456677 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.462531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.467365 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.468652 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.473726 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.482697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:22.494215 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:22.495594 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.496994 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.503599 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.512343 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.522066 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.523835 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.529654 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.530303 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.537787 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.542171 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.545248 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.555029 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.559077 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.564805 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.565397 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.582631 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.588092 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:22.601156 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:22.616479 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:22.632112 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:22.635614 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:22.635759 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:22.648046 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:22.675529 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.676991 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.687716 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.690670 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.697715 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.703783 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.708937 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.797323 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.797474 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.798182 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.803967 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.820648 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.822424 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.829895 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.830147 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:22.833395 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:22.840639 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:22.842872 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:22.847518 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:22.872644 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:22.874648 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:22.879379 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:22.883426 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:22.902302 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.968928 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:22.987164 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:22.991348 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:22.994035 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.025868 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.039270 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.046145 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.050335 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.070448 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.077853 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.077877 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.093896 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.097817 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.109692 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.131016 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.133975 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.144426 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.145099 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.146518 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.153018 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.153841 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:23.172076 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:23.176779 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.191478 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.204849 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.211372 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.216223 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.220589 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.232074 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.235167 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.245684 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:23.274222 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:23.343294 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.344686 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.353096 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.357038 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.363989 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.366468 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.369453 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:23.377511 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:23.389647 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.406912 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.409164 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.420490 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.424108 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.447046 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.448737 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:23.506419 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:23.545142 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.552158 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:23.554212 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:23.561750 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.565066 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.578796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.579095 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.587568 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.592126 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.598155 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.605401 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.634141 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.650404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.656857 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.657094 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.663918 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.665212 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.683370 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.704936 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.706611 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.708796 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.737641 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.770316 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.773931 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.774008 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.774760 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.779090 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.779695 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:23.807168 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:23.822645 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.871659 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.882323 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.887403 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.892479 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.901095 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.908859 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:23.911306 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.918602 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.925544 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:23.940172 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:23.943697 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:23.958238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:23.960692 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:23.967048 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.026288 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.026983 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.058259 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.062456 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.065936 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:24.073383 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:24.078831 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.083303 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:24.092491 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:24.141978 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.146024 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.154067 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.155890 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.178516 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.179986 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:24.183669 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:24.189878 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.197832 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.226143 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:24.234738 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:24.255177 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.257411 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.266763 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.274756 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.277227 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.290776 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.292781 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.296316 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.318691 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.321524 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.334053 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.340437 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.344508 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.346006 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.346729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:24.346963 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:24.351707 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.367898 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.383841 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.400913 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.472792 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.476219 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.481400 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.484952 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.497193 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.504232 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:24.506953 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:24.525889 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:24.550563 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 08:03:24.568645 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.570818 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.575277 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.585220 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.586560 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.586711 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.610770 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.614327 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.623015 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.625820 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.626271 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.629751 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.634312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.652838 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:24.679538 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:24.778155 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.785020 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.797237 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.809718 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.811094 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.811218 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.812731 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.833694 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.833952 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.835335 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:24.847218 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:24.851711 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.861850 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.870858 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.875514 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.879055 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:24.883790 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.885727 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.888620 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.889175 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.890012 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.923350 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.944547 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:24.948211 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:24.955057 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:24.962590 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:24.963399 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:24.974406 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.003872 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.010133 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:25.056353 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:25.057471 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.058373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.068257 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.070735 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:25.076186 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.085052 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.087030 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.093441 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.178769 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:25.189920 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.193280 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.197034 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.203397 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:25.204815 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:25.205849 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.220528 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.223341 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.227670 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:25.230088 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:25.231204 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.235529 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.248209 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.251969 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.269371 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.283047 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.398286 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.414584 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:25.417889 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:25.420135 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.432732 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.437628 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.448845 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.454167 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.454540 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.459839 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.462018 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.465296 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.466338 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.478821 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:25.484823 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.492825 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.495589 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.503129 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.504779 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.504952 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.507431 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.509231 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.509476 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.533923 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.544263 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.544929 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.547384 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.559969 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.563235 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.566658 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.571156 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.588557 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:25.624357 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:25.697994 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:25.709239 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.711891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:25.719398 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.732057 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.733953 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.734570 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.737815 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.751873 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.782174 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.792255 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.793083 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:25.793809 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:25.800822 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.809252 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.826498 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:25.827136 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.827685 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.835528 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:25.853466 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:25.854468 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.855974 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:25.883399 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:25.898614 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:25.993466 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:26.003881 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:26.009860 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.010793 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.013353 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:26.024356 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:26.026314 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.040997 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.045027 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.052758 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.053179 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.063944 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.065346 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.066888 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.081004 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.089316 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.090090 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.091781 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.102270 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.111344 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.116691 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.119052 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.135846 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.193524 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.194902 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.197021 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.225688 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.226218 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.227792 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.254835 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.346817 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.364836 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.365078 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.376286 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:26.379312 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:03:26.379909 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:26.379981 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:26.391567 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.393913 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.400782 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.402222 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.407382 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.413060 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.421837 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.425361 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.426544 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:26.550273 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:26.573886 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.587927 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.589601 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:26.600960 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:26.616410 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.622572 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.622825 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.626376 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.626863 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.632947 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.633756 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.651054 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.653591 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.689210 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.694577 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.697047 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.698958 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.705227 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:26.714260 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:26.715386 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.721504 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.739506 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.742002 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.750350 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.755362 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.757889 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.761137 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.771339 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:26.779347 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:26.780877 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.787900 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.848268 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.852219 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.852216 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.856153 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.858871 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.869594 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.878278 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:26.940258 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:26.940944 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:26.943022 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:26.950196 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:26.955003 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:26.956189 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:26.956616 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:26.957784 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.041869 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.052889 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.054883 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.055821 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.056339 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.075385 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.083583 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:27.128155 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:27.129822 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:27.149515 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.158079 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.164249 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.165474 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.166501 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.177054 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.194741 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.197108 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.198383 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:27.214402 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:27.216075 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.223081 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.250520 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.252195 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.257206 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:27.263526 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:27.264175 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:27.313762 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.313920 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.322329 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.325212 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:27.340463 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:27.342175 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.347827 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.349125 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.366235 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.378284 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.389847 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.407414 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.408790 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.411036 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.419649 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.480151 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.484405 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.484426 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.489918 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.491393 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.492197 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.509852 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:27.528802 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:27.601897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.603997 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.604782 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.606684 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:27.611011 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:27.612380 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.631397 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.634397 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.641752 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.646099 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.648762 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.662482 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.663322 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.663620 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.666082 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:27.685144 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.713207 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:27.716610 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.720562 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.720815 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.733853 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.741106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.745406 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.760010 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:27.788067 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:27.861733 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:27.868048 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:27.877662 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.881510 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.887544 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.891990 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.892150 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.905414 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:27.912523 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.918813 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:27.929983 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:27.934486 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.939081 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:27.945842 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:27.969273 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:27.995193 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.001617 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.004509 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.004565 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.015964 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.035063 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.062673 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.069577 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.069677 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.076987 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.077427 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.084288 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.095468 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.095885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:28.113693 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.117295 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.184633 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.197924 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:28.200943 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.201625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.203940 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.213295 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.225538 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.248445 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.276855 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.293394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.299695 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.308558 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.313131 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:28.328253 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:28.332116 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.335542 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:28.337303 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:28.347376 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.354532 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.357755 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.377338 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.385366 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.392663 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:28.393591 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:28.400125 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.413594 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:28.422264 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:28.465608 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.469457 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.470032 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.470568 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.484592 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.484653 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.511631 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.564270 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.568807 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.578330 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.579767 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.590137 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.595431 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.596751 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:28.653668 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 08:03:28.670672 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.676267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.677866 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.679780 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.690734 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.692063 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.692669 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.698912 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.707000 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.716207 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.744663 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.766795 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.770447 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.771685 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.791865 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.796715 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:28.800133 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:28.803213 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.808212 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.810047 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:28.810268 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:28.818464 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.824944 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.872778 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:28.877753 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:28.883780 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:28.884380 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:28.898296 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:28.904796 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:28.906409 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:28.927181 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:29.025547 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.030719 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.032058 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.049054 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.052809 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.053689 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.063876 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.068279 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.076887 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.087064 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.094115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.096925 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.099057 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.108541 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.117229 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.119397 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.125521 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.130381 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.148962 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.152748 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.165984 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:29.297729 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:29.309388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.324625 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.325562 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.332572 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:29.333199 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:29.333750 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.334553 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.340827 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:29.346498 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:29.354283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.355031 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.373064 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.377000 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.383317 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.416664 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.416919 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.422387 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.432152 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.437644 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.439649 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.441221 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.447420 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.447579 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.459347 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.460822 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.483402 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.487123 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:29.503401 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 08:03:29.514966 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:29.518527 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:29.523689 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.536530 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.646322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.653182 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.658200 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.659598 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.661464 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.662967 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.664705 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.667506 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.669261 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.699785 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.701004 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.701347 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:29.720357 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:29.721987 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.723749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.727437 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.731496 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.731745 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:29.752079 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:29.757491 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.759374 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.766538 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.772277 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:29.880555 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:29.903338 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:29.905529 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.906033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.909551 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.910738 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.923666 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.929222 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:29.958163 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:29.960232 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:29.960428 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:29.960702 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.961498 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:29.966732 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:29.970993 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:29.982561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.007742 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:30.012253 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.014380 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.029014 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.043035 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.044384 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.046548 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.049715 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.154805 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.155797 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:30.156803 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.166563 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.175886 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.177700 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.178719 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.184031 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:30.229149 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:30.242228 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.244449 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:30.245019 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:30.248414 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.264057 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.268544 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.272230 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.300937 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.323705 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.326699 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.336117 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:30.340876 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:30.343914 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.349862 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:30.350986 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:30.351342 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.364149 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.390303 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.392738 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.397786 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.406312 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.407224 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.425385 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:30.436033 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.442013 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.475793 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.477187 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.480463 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.491580 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.491801 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.497274 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.521819 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.601477 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.603780 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.613999 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.616495 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.626757 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.627151 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.627833 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:30.635600 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.644510 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.657997 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:30.668852 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.676352 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.678122 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.678367 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.679039 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.695021 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.708310 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:30.736505 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:30.741202 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.745056 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.751649 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.751878 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.756209 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.757947 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.831120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.838194 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.838979 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.839893 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.841018 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.842123 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.845456 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:30.867291 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:30.887269 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:30.916539 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:30.917312 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:30.919725 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.921665 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:30.926777 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:30.930191 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.945957 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:30.991196 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:30.992322 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.000640 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.014075 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.014974 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:31.020849 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:31.026381 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.027937 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.119205 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.119715 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.123159 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.126895 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.128475 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.141383 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.155450 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:31.206192 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:31.219528 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.220325 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.226200 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.228085 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.232497 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.234395 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.256981 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.265908 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.270221 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.281162 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.282671 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.283681 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.296198 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.311141 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:31.323818 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:31.411092 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 08:03:31.411729 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:31.414356 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.422701 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.425046 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.425439 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.431411 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.435946 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.439193 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.447039 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.462623 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:31.469784 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:31.486757 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.495594 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.499723 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.507968 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:31.515723 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:31.517817 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.542182 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.549861 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.551324 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.552979 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.561312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.566802 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.582075 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.627104 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.630875 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.636786 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.637276 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.662786 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.671885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.673014 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.703935 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.706602 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.718863 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.719132 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.729956 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.737365 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.739890 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.784348 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.784606 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.787078 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.793143 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.805533 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:31.816264 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:31.818773 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:31.819381 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 08:03:31.819782 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.911980 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.925012 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.937557 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:31.938373 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:31.938637 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:31.951333 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.959283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.974440 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.992283 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:31.995322 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:31.995534 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:31.996780 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.013911 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.039129 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.069264 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.072441 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.074101 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.074734 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.086668 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.098857 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.099539 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:32.127451 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:32.154131 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:32.197932 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.201735 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.204660 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.204731 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.213428 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.230068 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.231628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.235191 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.245225 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.246867 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.260330 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.261296 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.262359 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:32.301919 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:32.303436 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.328815 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:32.330963 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:32.333483 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.343326 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.343570 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.366281 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.368927 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.386904 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:32.421501 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:32.431121 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:32.459336 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.465621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.468050 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.470324 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.477324 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.504850 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:32.515816 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:32.516729 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:32.520068 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:32.520355 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:32.529846 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:32.533043 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.537549 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.552929 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.554220 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.554874 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.563949 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.598528 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.603005 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.603346 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.610842 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.610947 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.626713 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.633136 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.640848 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.698511 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.704422 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.707906 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.707947 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.711014 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.713469 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.722390 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.772867 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.774582 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.774967 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:32.782792 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.799722 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.800206 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.800998 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:32.809123 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:32.863647 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:32.919572 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.921906 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.952129 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.952899 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:32.956830 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:32.970633 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.972965 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:32.973894 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:32.975569 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:32.982740 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:33.002171 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:33.003469 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.009287 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.014983 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.040431 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:33.040581 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:33.046489 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.047979 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.054001 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.059707 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.079872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.092075 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.104110 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.160667 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.161877 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.162258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.173868 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.175099 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.187837 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.198936 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.206421 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.207046 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.209477 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.211784 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.212440 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.215390 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.241880 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:33.291096 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:33.323721 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.327734 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.327878 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.339082 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.343559 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.351459 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.357079 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:33.366668 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.429244 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:33.458317 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.462661 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.477140 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.481117 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.498589 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.503264 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.506657 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.515141 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.529282 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.530154 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.537513 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.557806 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.562058 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.563978 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:33.568626 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:33.569192 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.570959 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.581575 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.599710 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.602605 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.612483 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.614199 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:33.627812 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:33.660596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.662179 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.671361 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.677107 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.681521 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.685433 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.704093 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:33.725416 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 08:03:33.726053 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:33.744802 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.768146 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:33.770879 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.777817 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.780681 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.781235 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.785048 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.795413 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.800896 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:33.811496 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:33.848592 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.853935 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.857950 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.860356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:33.874078 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.886251 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.886317 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.937741 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:33.943084 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:33.945626 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:33.947572 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:33.949402 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:33.955101 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:33.977538 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:34.007995 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.069171 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:34.077431 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:34.078999 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.086122 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.093991 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.102403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.109660 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.111807 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.114010 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.121801 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.124650 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.126019 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.151647 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.168694 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.175344 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.189893 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.190593 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.195333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.199255 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.211702 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.212704 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.225508 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.310954 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.316386 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.317231 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.317986 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.323007 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.339942 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:34.358492 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:34.391443 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.425056 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.427047 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.428354 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.452838 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.452929 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.462126 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:34.464391 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 08:03:34.464727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.473161 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.475492 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.485028 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.491815 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.493490 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.494328 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.498675 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.510902 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.511819 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.522816 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.528327 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.529161 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.538034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:34.554082 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:34.554606 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:34.636799 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.649827 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:34.657858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.664546 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.674420 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.677276 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.682984 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.707238 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.724258 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.739341 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.744935 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.745632 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.756527 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.759097 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.763832 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.765113 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:34.765833 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.775691 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.780722 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.785434 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:34.792932 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:34.803115 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.831349 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.832087 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:34.875888 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:34.912961 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:34.915287 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:34.925505 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:34.925618 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:34.927835 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.938478 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:34.943246 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:34.984989 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.986224 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:35.044198 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.046264 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.055984 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.056275 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.063832 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.078039 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.086885 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.092597 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.097313 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.118909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.119822 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.124984 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.126004 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.137132 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.137151 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.153371 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:35.168660 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:35.180868 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.191991 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.193696 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.203843 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.214437 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:35.215821 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:35.316814 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.321979 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.329205 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.349670 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.352885 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:35.358855 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:35.371696 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.377238 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.381084 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.382710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.383935 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.395010 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.414366 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.415992 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.427852 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.462522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.465035 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.465987 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.469456 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.471100 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.486766 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.525193 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.545879 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:35.549539 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:35.556771 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.557478 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.562352 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.568976 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.569886 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.578684 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.602515 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.602515 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.604790 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.614729 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.618165 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:35.619104 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 08:03:35.623879 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.632714 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:35.703529 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:35.728756 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:35.735162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.737501 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.744993 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.747483 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.752765 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.778219 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.783109 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.783349 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:35.783576 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:35.798586 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:35.811378 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.817378 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.826215 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.829164 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.843752 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.907466 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:35.908234 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:35.908226 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.908865 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:35.909780 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:35.930440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:35.937539 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:35.956350 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:35.972896 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:36.028391 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.037562 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.038576 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.049188 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.053658 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.064944 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.068167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.074013 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.076366 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.083505 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.112627 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.113456 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.114780 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.118078 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.118948 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.125599 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.127103 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.140707 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.145087 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.153724 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:36.182297 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:36.187316 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:36.220115 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:36.235597 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:36.311628 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.312160 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.313916 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.314209 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.319209 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.323592 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.338179 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.344252 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.351032 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:36.351791 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:36.359198 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:36.361402 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.362666 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.364102 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.375330 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.377480 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.399771 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.407944 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.427692 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.434475 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.439092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.439999 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:36.441928 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:36.451653 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.524859 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.536178 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.540796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.542969 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.544773 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.545452 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.546767 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.591501 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.599624 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:36.602969 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:03:36.603974 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:36.608958 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.609225 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.610649 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.610785 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.645195 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.677362 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.677464 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.680331 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.685564 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.686305 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.692310 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.702690 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:36.716845 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:36.766213 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:36.811317 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.827988 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.829891 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.840119 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.848157 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.860674 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.863090 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:36.873227 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:36.894860 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.914391 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.916253 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:36.919400 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.923655 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:36.927469 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:36.941584 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.947096 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:36.957399 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.959903 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.963115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:36.968015 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:36.983953 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.016808 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.024439 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.035265 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.040635 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.042277 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.043646 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.043687 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.085997 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.112068 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.119014 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.129762 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.136488 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.140627 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:37.142794 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:37.148032 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.168880 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:37.169461 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:03:37.262724 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.274708 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.275535 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.276772 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.283519 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.291606 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.324707 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.331866 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.337201 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:37.340111 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:37.360507 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.383663 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:37.385043 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:37.396569 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.397560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.404582 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.413611 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.420194 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.423651 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.424153 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.435164 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.444888 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.451201 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.452129 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:37.456326 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:37.470333 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:37.479269 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.484885 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.491794 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.495676 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.507938 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.513208 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:37.565584 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:37.589273 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:37.613546 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:03:37.633987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.640642 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.640754 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.642678 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.644338 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.646109 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.674134 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.684551 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.692523 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.697759 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.710959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.712904 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.724130 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:37.731849 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.736845 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.742959 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.755893 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.759405 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:37.770670 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:37.776693 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.777598 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.793177 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:37.840627 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:37.890655 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:37.893193 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:37.902989 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:37.903520 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.901723 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:37.912802 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:37.942083 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:37.952618 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:37.984225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:37.993696 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:37.996366 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.014096 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.019297 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.022655 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.028398 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:38.030598 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:38.054201 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.060317 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.068876 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.069638 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.072688 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.083269 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.089044 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.092913 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.094411 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.109447 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.117029 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.120435 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.141903 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.145694 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.165755 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.252868 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.256222 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.256907 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.263481 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.266983 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.283985 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.286170 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:38.323910 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:38.357594 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.363056 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.371042 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.380792 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.391187 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.396908 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.414690 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.430509 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:38.431201 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:38.441089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.451175 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.453899 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.462752 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.471709 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.471962 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.480541 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.483062 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.486039 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.495643 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.500574 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.517720 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.519373 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:38.547133 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:38.591526 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.592343 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.594842 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.602788 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.607618 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.631232 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.633976 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:38.658774 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:38.673475 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.681324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.682371 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.683382 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.684893 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.703181 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.714010 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:38.727620 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:38.801120 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.823266 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.824795 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.825815 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.829187 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:38.837356 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:38.837394 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:38.838274 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:38.847105 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.848289 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:38.851377 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:38.869540 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:38.872015 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:38.873226 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.873992 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.875258 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:38.876219 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:38.893872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.903283 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.913797 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:38.923231 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.925554 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:38.937851 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:38.939177 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:38.952719 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:38.968206 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:38.972028 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:39.066421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.072789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.074074 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.075361 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.075495 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.093572 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.094517 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:39.141415 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:39.188050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.189571 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.194747 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.202558 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:39.209736 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 08:03:39.215485 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:39.232714 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:39.251096 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.259238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.261265 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.263288 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.265362 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.271050 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.271056 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.273614 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:39.284399 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:39.285161 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.286504 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.293025 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.311527 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.311816 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.312826 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.313297 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.320006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.332681 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.357790 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.360517 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.366230 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.373499 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.378038 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.399662 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.492060 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.492794 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.495324 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.503573 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.513177 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.518122 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.545704 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:39.556949 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:39.593503 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:39.605588 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.609351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.610278 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.610980 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.614341 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.615597 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.623872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.645669 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.650993 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:39.652839 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:39.655184 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.657440 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:39.663867 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:39.672128 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.678617 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.679911 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.811070 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.825039 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.849917 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.850335 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.854534 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.859619 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.862993 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.884804 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.896947 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.897529 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.899493 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:39.908815 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.923960 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:39.928231 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.935551 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:39.946998 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.948415 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:39.953305 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.958797 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:39.965623 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:39.966106 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.975699 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:39.979481 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:39.983971 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.006203 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.006523 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.017159 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.058295 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.065409 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:40.068756 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:40.070817 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.073328 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.090728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.092612 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.106194 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.106441 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.108633 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:40.226511 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:40.230777 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:40.232550 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:40.233788 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:40.234714 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.240870 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.254413 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.269675 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.279081 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.280359 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.281853 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.291162 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.292428 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.296660 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.302166 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.327924 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.330099 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.345103 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.360941 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.361361 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.361943 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.364106 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:40.366715 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:40.375041 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.392922 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.519135 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.525225 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.529237 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.547825 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.561517 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.564585 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.571017 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.573856 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.586061 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.588289 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.606267 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.615942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.616662 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:40.621670 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:40.624866 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.631109 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.634850 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.635051 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.640316 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.656647 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.656829 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.670279 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:40.690208 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:40.722697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.723973 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:40.735117 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:40.739015 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.741675 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.743050 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.760295 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.767820 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.809581 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.813704 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.815630 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.824820 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:40.830161 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.843019 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.845201 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:40.888362 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:40.947242 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:40.952483 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.955545 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:40.963928 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.968267 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:40.972090 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:40.986128 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:40.995306 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.010267 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.011605 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.022380 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.030419 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.037595 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.053170 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:41.061749 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:41.107936 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:41.133546 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:41.143467 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.147639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.158741 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.159366 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.181257 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:41.185817 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:41.200060 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.200862 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.209105 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.211355 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.222827 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:41.232560 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:41.238659 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.252408 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:41.260386 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:41.265291 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.267956 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.273458 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.278017 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.279225 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.286051 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.286623 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.309214 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.309875 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:41.367048 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:41.434360 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.438421 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.451691 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.454142 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.455394 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.463443 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.476509 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.543318 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.543665 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.544582 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.545706 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.545740 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.561662 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:41.579780 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:41.593731 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.598964 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.603073 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.611001 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.611649 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.611812 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.613467 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:41.614189 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:03:41.622997 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:41.634358 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.636915 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.654897 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.655805 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.660230 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.674479 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.683943 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:41.691727 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:41.703563 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:41.709873 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.796747 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.802369 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.805919 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.807001 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.823463 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.843448 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.864957 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.869136 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.873518 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.876165 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.877663 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.885569 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.889214 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:41.892699 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.900076 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:41.904677 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:41.919063 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:41.931702 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.946157 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.946718 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:41.971882 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:41.998289 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.004747 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.007102 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.008978 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.011814 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.020573 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.027019 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:42.075075 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:42.119486 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.125741 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.129173 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.129762 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.136651 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:42.141947 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:42.144883 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.146947 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:42.146993 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:42.199533 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:03:42.222560 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:42.228622 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.234103 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.236075 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.240134 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.262231 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.267798 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.273939 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.277755 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.280100 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.301707 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.327122 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.328015 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.343807 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.355674 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:42.382174 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:42.405616 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.420424 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.429097 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.434576 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.439293 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.439832 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.458511 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.462018 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.464578 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.466535 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.492120 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.500243 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.504802 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.511937 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:42.530025 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:42.590034 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.593605 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.599261 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.604409 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.605830 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.616907 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.624950 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.631762 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.635742 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.640460 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.655084 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.659199 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.659881 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:42.675566 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:03:42.681116 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:42.689620 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:42.721133 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:42.733579 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:42.736539 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:42.739840 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:42.741770 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.749768 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.761338 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.761998 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.765627 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.767071 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.767660 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.781602 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.782212 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.796051 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.809608 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.841679 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.841928 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.916965 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.919860 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.921046 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:42.923239 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:42.923265 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:42.926046 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:42.935772 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.995767 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:42.996006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.996093 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:42.996696 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:43.002234 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:43.011862 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.018149 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.030782 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.070837 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.075188 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.078239 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.093000 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:43.093079 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:43.093433 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.093791 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.098282 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.195402 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.203420 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.217679 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.244889 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.247581 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.252722 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:43.261626 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:43.262320 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.263489 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.267946 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.271149 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.274966 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.283123 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.285709 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.294039 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.302665 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.304929 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.313403 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.325196 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:43.326534 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:43.336066 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.340049 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:43.350240 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:43.363406 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.470211 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.476613 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.477073 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.485788 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.486348 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.487447 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.509598 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.543988 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.550472 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.556528 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.561287 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.571628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.589380 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.591715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.596163 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.598028 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.598266 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.607092 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.615504 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:43.627235 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:43.627964 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.658720 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.736830 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.742964 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.750574 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.754315 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.754778 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.760097 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.761970 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.763969 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:43.764643 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:03:43.766239 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.785576 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:43.789196 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:43.789736 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:43.798267 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:43.822818 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.830910 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:43.832782 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:43.867754 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:43.877004 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.881989 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:43.885612 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:43.895164 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.895812 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.901371 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.903836 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:43.910135 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:43.911564 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.973232 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:43.975049 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:43.975669 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:43.979876 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:43.982013 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:43.988684 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.014235 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:44.030285 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:44.096011 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.097242 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.100169 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.105785 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.111994 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.121811 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.124637 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.128382 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.135316 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.146363 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.148537 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.152673 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.180811 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.187428 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:44.254622 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:44.257817 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.259109 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.270763 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.284260 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.284929 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.300225 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.318086 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.365366 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.366840 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.368083 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.368207 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.369170 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.376688 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.409729 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:44.449339 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:44.453687 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:44.492926 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.497283 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.497457 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.502875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.507875 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.508433 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.520716 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.521982 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.522797 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.524196 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.526172 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.550346 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.555231 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.585650 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.590464 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:44.594259 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 08:03:44.606482 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.612618 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.616991 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.619361 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.624405 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.644801 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.657819 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.663360 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:44.683702 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:44.688565 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.689038 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.690102 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.698371 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.700803 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:44.702126 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:44.789388 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:44.790760 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.796203 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.803820 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.814268 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.817456 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.839502 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.840639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.841292 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.844084 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:44.848276 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:44.848780 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.862777 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.872521 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.873590 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.879109 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:44.936180 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:44.948864 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:44.951884 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:44.957013 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.971208 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:44.987504 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:44.996265 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:45.050372 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:45.072821 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.081154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:45.082284 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:45.085242 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.092203 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.100553 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.100554 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.101648 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.101804 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.106041 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.108480 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.123014 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.126256 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.131839 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.157450 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:45.180536 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:45.228649 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.234768 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.235278 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.243130 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.270727 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.271054 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.271400 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.295064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.309626 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.314486 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.336040 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.340135 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.342283 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.342711 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:45.366312 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:45.401535 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.410212 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.411652 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.420840 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.424777 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.444883 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.445479 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:45.454007 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:45.497156 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.505034 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.518774 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.524502 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.533864 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.534798 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.558216 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.569392 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.571810 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.572188 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.591787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.599929 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.600617 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.602248 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.625499 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.631976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.633723 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.635304 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.643114 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.646214 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.651435 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:45.652112 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:45.749785 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.752367 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:45.762362 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:45.763012 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.766283 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.774101 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.774971 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.809222 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.812500 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:45.815677 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:45.817214 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.828050 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.830092 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:45.830715 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.832917 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:45.882742 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:45.952178 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:45.959347 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:45.961591 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:45.962666 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.004361 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.009053 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.009342 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:46.017699 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:46.024482 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.026955 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.027256 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.027333 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.030743 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.039212 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.050300 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.073207 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:46.074859 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:46.076860 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:46.084584 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:46.089409 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.090440 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.104629 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.106889 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.107900 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.127384 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.165448 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.173858 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.177469 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.181638 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.198828 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.200481 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.216467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.302613 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.304375 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.322602 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.323520 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.324800 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:46.327250 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:46.328181 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:46.333240 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.345930 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:46.364006 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:46.366992 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.367225 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.373392 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.376775 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.401301 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.434497 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.436909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.438341 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.439952 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.444465 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:46.461012 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:03:46.461772 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:46.467298 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.479817 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.483971 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.486092 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.570736 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.576015 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.579064 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.583382 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.585996 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.612139 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:46.614740 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:03:46.615841 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:46.616451 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:46.623430 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:46.629530 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.633626 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.645090 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.645835 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.646710 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.647089 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.659826 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.725303 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.732753 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.732825 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.732915 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.743465 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.744485 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.746853 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.801090 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.803028 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.807365 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.813807 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.814322 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.822268 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.839877 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:46.882810 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:46.890154 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:46.891345 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:46.891731 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:46.892874 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:46.906232 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:46.906978 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:46.908214 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.001343 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.006842 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.019680 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.030225 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.034637 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.041183 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.046007 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:47.047813 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:47.091392 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:47.100098 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 08:03:47.105203 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.111722 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.121654 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.129937 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.141757 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:47.160804 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:47.184744 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:47.188154 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:47.191005 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.191092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.205404 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.209551 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.209560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.209997 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.223788 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.225714 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.230322 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.232481 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.240379 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.263911 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.271071 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:47.278800 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:47.282694 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.282995 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.289541 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.293745 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.303599 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.308804 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.330852 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.363685 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:47.395319 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:47.416766 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.421092 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.428419 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.432483 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.433501 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.433557 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.448498 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.504098 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.507810 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.508158 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.511436 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.519491 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.531836 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.532996 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:47.551430 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:47.624172 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.631944 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.634166 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.634166 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.639834 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.645066 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.670081 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.703758 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.705224 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.738411 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.739346 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.745008 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.746897 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.747628 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.760825 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.766335 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.768280 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.773902 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.774672 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:47.782104 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:03:47.782999 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:47.784231 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.792701 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:47.799974 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:47.840307 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:47.869405 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.869566 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.873797 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.879056 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.883250 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:47.889148 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.902736 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:47.912642 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:47.947261 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:47.957291 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:47.961932 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:47.963004 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:47.976197 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:47.986310 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.005679 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.056190 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:48.078167 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.089423 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.095484 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.095727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.096699 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.098604 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:48.131583 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:48.145964 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.182729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.191089 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.202225 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.207883 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.218416 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.219882 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.224200 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:48.240922 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.285305 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.285302 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.291431 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.299998 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.301496 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.308573 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.321965 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.327653 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.333209 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.351158 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.354724 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.360356 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.360458 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:48.379134 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:48.397996 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.424773 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 08:03:48.440598 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.444373 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.456113 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:48.456414 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.456815 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.458660 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.464998 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.476487 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.482880 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.493367 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.496348 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.509474 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.517047 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.520196 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.523075 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:48.611829 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:48.630898 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.643025 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:48.645732 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.662395 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.664320 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.664737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.667264 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.673856 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.689702 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.694082 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.699379 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.707624 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.710191 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.713708 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.739068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.762169 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.846202 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.851433 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.857786 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.863297 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:48.869582 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.872771 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.884943 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:48.891264 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:48.895238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:48.907044 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:48.914240 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:48.918805 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:03:48.920394 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.923399 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:48.928418 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:48.933789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:48.939082 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:48.953295 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:49.015640 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.018272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.018836 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.024958 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.050968 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.059244 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.061012 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.067162 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.070132 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.072103 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:49.079620 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:49.080247 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.087639 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.099236 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.101838 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.142406 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.144269 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.145680 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.146988 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.148803 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.151875 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.152374 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:49.158082 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:49.207105 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:49.270339 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.270462 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.275825 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.291072 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.295173 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.301321 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.342631 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.362092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.368567 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.374805 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.376310 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.376787 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:49.391394 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:49.392551 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.393625 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:49.405499 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:49.406625 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.415269 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.420763 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.430511 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.438699 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.457088 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.472505 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.495090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.498641 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.506121 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.518102 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.519704 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.531021 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.535187 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.617904 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.618264 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.620841 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.631814 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.643810 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.645350 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:49.646147 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 08:03:49.646593 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:49.659665 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:49.671007 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.673698 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.676636 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.685194 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.687983 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.690995 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.693760 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:49.705336 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:49.753566 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.759162 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.761139 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.764376 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.771776 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:49.793852 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:49.795722 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.798403 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.875319 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.875533 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:49.875623 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.881451 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.891584 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.896488 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:49.897889 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:49.968403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:49.980144 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:49.982620 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:49.986375 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.002377 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.011191 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.027967 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.033272 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.033995 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.035115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.036154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.036609 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.039324 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.073410 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:50.119051 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:50.132880 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.134521 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.148308 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.150363 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.150388 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.153020 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.166635 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:50.198449 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:50.227427 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.228129 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.231526 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.233873 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.235239 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.235643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.258808 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:50.288373 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:50.310528 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.315395 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.317914 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.324856 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.336453 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.349067 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.352442 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.393153 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.400724 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.402773 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:50.411087 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:50.429055 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.432662 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.440434 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.447700 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.489259 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.502102 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.507263 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.507990 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.526829 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:50.536067 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:50.550943 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.560075 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.571884 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.573624 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.580634 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.583468 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:50.595746 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:50.601935 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.606263 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.606582 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:50.646148 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 08:03:50.663465 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.665053 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.674327 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.676301 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.677657 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.710633 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.723075 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.730084 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:50.738399 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:50.759624 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.762245 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.767933 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.782013 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:50.785950 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.793818 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:50.797794 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.810777 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.817137 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:50.819136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:50.825659 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.827908 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:50.836112 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.836834 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.858256 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:50.980432 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:50.987201 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:50.995746 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.006968 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.009145 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.012759 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.029560 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.044319 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.053218 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.067528 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.087319 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.088447 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.088563 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.100966 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.101755 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.116689 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.117905 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.121433 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.133425 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.135061 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.143841 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.147312 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.155977 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.156466 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.156838 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.165602 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.168057 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.168833 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.182751 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.190862 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.193624 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.204401 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.209543 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.230499 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:51.232945 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:51.235747 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:51.236959 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:51.243737 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:51.260320 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:51.377310 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.380189 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.380511 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.387600 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.390738 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.393935 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.416042 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:51.423475 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:51.508036 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.514122 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.522022 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.531818 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.532109 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:51.538473 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:03:51.539541 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:51.539566 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:51.554347 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:51.556671 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.626657 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.630023 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.640955 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.648974 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.650257 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:51.656666 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:51.662971 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.667951 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.672668 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:51.673725 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:03:51.674472 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:51.682811 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.686326 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.687758 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.700637 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.710450 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.733944 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.761168 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.763804 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.768676 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.777312 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.786499 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:51.799333 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:51.808874 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.819121 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.848138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.856380 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.858460 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:51.859081 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.860214 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:51.867769 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:51.874814 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:51.893965 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.956743 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:51.969797 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:51.978343 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:51.987941 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:51.994874 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.007401 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.014360 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.020483 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.033807 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.034425 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.035966 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.044373 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.048831 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.060390 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:52.120383 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:52.145994 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.151911 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.154251 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.160183 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:52.160912 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:52.178506 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.178798 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.191411 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.278096 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.278491 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.294469 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.294605 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.296103 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.305395 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.311697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.316182 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.319300 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.321197 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:52.324139 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:52.329349 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.334380 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.344869 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.352713 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.355839 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.380774 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.389111 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.393262 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.397492 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.403148 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.431902 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.456187 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.457607 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.470105 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.472993 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.475317 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.491467 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:52.491487 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:52.499383 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:03:52.501273 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:52.525282 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:52.613930 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.620622 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.628533 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.629132 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.633303 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.649632 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.654062 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.654253 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.661330 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.669292 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.682910 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.687422 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.695955 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.706693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.711179 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.715008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.720312 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.720621 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.737577 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.772755 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.781458 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:52.790549 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:03:52.860522 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:52.868561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:52.869256 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:52.874501 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:52.900764 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:52.905232 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.907158 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.913962 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.919170 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:52.930828 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:52.968554 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:52.973968 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:52.986670 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:52.987917 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:52.988050 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.001004 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.003019 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.006160 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.012147 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.014461 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.019855 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.034416 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:53.038314 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:53.044312 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:53.056557 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:53.077303 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.080053 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.095532 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.096042 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.097312 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.103786 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.123636 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.138860 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.167850 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.168879 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.171840 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.181834 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.182523 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.191605 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.192960 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:53.311570 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:53.320896 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.327528 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.329607 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.334287 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:53.351428 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:53.358418 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.361713 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.361902 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.371163 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.375335 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.386621 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.387820 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.390062 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.391819 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.406586 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.416440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:53.421725 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:53.430044 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:53.437306 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.437566 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.441604 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.443764 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.447851 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.470075 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.606500 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.607853 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.623344 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:53.623545 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:53.624114 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.631860 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.648267 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.652076 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.662971 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.666978 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.667308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.667428 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.667946 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.676057 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.682296 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.682849 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.688491 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.689882 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.695068 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.710901 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.728057 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.736135 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:53.741909 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:53.743012 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.750000 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.762542 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.768651 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.786471 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.803517 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:53.809115 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:53.812175 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:53.817701 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:53.880787 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:53.883894 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.884478 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.894340 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.898777 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.919135 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:53.921104 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:53.924921 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.956337 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:53.956619 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:53.966192 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.975917 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:53.977546 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:53.977718 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:53.993108 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:54.010970 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:54.061980 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:54.116874 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:03:54.137421 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.155468 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.156524 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.161777 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.164614 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.166191 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.166734 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.183995 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:54.184902 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:54.192170 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.204388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.212013 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.213410 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.219233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.219468 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.227393 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.228299 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.235474 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.239130 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.245361 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.278213 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.283419 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:54.314761 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:54.381850 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.383239 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.386698 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.389684 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.391933 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:54.395038 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:54.400823 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.407661 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:54.416020 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:54.461571 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.462028 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.473129 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.479781 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.501032 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.511137 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.513267 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.515828 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.526556 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.529640 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.539860 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.540891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.541155 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:54.544112 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:54.547348 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.601606 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.612285 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.619904 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.623977 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.628109 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.631978 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.635116 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.776074 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.789748 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.795284 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.798995 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.805554 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.806838 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.807861 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.809232 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.828896 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.829883 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.837286 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.844954 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.853838 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:54.854789 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:54.856213 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:54.856582 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.864564 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.874197 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.875646 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.892547 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.900794 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.902503 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:54.906741 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.956315 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:54.962551 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:54.966692 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:54.975261 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:54.981501 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:54.991066 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:55.015334 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:55.015520 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:55.035053 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:55.077068 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.077097 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.078027 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.078976 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.082815 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.085150 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.092121 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.152163 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.153972 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.154719 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.156828 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:55.164484 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:55.171098 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.182078 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.196234 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:55.273222 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:55.303364 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:55.334992 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:55.336763 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.345969 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.355366 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.359567 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.366184 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.368985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.371480 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.378459 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.388669 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.390463 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.392784 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.401882 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.409355 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.415441 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.420312 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.426947 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.445288 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:55.447381 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:55.452220 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.473513 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.473781 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.485364 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.485929 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.489140 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:55.500269 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:55.509996 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.515134 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.520628 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.523239 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:55.536581 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:55.584378 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.592144 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.595728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.596776 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.605408 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.622222 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.625378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.699939 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.700673 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.703238 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.703994 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:55.731336 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.761899 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.763787 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.765273 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.774132 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.774284 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.793002 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.796110 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.806762 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:55.838371 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:55.848028 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.865024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:55.872118 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.874115 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:55.881165 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:55.888268 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:55.897486 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:55.915620 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:55.934393 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.941926 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:55.999136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.009317 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.017055 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.032030 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.033632 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.034300 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.039748 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:56.047860 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:56.049374 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:56.062237 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:56.067611 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:56.085292 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.086892 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.094332 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.105573 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.105741 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.115397 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.118001 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.125931 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.127120 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.127583 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.147974 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.157255 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.161360 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.274550 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.277454 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.280360 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.282634 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.292645 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.302349 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.302431 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.302675 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.306205 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:56.317073 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:56.326707 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.338478 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.361038 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.368537 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.370729 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:56.381322 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:56.382179 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:56.410828 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.421372 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.427767 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.430580 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.430639 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.432065 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.434841 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:56.549654 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:56.551452 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.560842 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.562685 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.576757 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.579928 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.608520 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.616623 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.633242 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.641396 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.648140 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.653134 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.653438 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.654160 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.654512 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.655907 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.658396 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.673029 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:56.683326 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 08:03:56.694679 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.703776 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:56.710984 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:56.719415 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.738693 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.745643 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.746620 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.749450 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.749881 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.763366 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.768280 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.772454 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:56.800249 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:56.875324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.877863 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:56.878512 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:56.878758 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.885529 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.896323 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:56.900363 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:03:56.908510 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.910454 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:56.911319 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:56.914045 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:56.917855 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:56.919298 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:56.928267 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:56.929849 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:56.931432 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:56.939662 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:03:56.951510 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:57.057318 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.065143 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.076321 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.081964 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.086851 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.092815 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.107442 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:57.132393 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 08:03:57.166019 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.171923 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.175894 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.183384 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.186333 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.196027 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.223598 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.224395 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.230366 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.235090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.240741 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.249135 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.255093 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.259071 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.265769 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.270325 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.281595 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.293114 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.293294 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.301606 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.320887 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:57.356665 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:57.367365 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:57.370090 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:57.402156 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.402772 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.408687 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.409803 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.422089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.422758 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.427959 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.543433 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.545400 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.545583 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.549724 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.553165 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.563431 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.581957 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.592419 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.593340 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.602332 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.613543 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.619801 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.621557 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.623294 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.639081 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.642010 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.647994 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.657825 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.659384 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.677624 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:57.680383 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:57.701619 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.723524 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.725770 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.727308 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.727321 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.729234 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.743913 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.760131 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:57.794126 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:03:57.794859 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:57.804347 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.808880 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.810772 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.817145 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.822660 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:57.824347 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:57.830734 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.834869 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:57.900187 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:57.917693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.919306 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:57.923703 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.928028 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.949919 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.958964 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.976591 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:57.985755 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:57.986806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:57.988034 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:57.994853 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:57.996543 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.020685 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.036214 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:58.155992 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:03:58.171105 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:58.174515 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.175942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.179737 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.184387 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.193451 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.216060 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.220981 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.224570 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.250838 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.260321 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.268329 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.277549 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.289419 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.296447 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.308641 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.315138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.318310 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.319656 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.323027 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.326195 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.331739 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:58.337827 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:03:58.356366 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.357362 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.360996 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.363492 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.375727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.377454 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.399871 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:58.456773 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:58.456859 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:58.472869 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:58.478322 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:58.494310 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.495254 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.503324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.504346 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.514433 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.519541 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.536679 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.574882 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.577911 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.586750 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.587182 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.603125 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.606251 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.608450 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.610162 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:58.612133 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:58.614504 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.623591 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.636941 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.640417 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.643245 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.673688 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.756331 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.773118 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.773957 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.776950 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.781584 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.799559 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:03:58.800176 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:58.815783 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.844130 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.844146 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.846616 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.855000 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.856209 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:58.856909 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.857256 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.861613 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.876682 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.882550 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.893527 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.914597 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:58.925017 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:58.932262 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:58.942430 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:58.950150 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.969015 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:58.973699 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:58.973950 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.976002 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:58.976609 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:58.984925 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:58.999466 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:59.024132 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:59.046404 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:03:59.099105 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.108610 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.114178 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.116976 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.117516 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:59.123109 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:59.125450 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.126019 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.131085 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.141423 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.142570 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:59.151135 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 08:03:59.164067 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.166989 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.170213 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.185174 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.307894 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.310984 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.319073 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.337510 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.340858 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.369864 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.384183 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:59.399883 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:03:59.407249 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:59.407523 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.418823 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.420838 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.438278 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.439607 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.440356 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.440940 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.442713 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.446244 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.448393 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.461582 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.467458 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.480607 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.490211 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.503096 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.503481 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.528551 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.531561 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.553663 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.557366 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.561146 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.575625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:03:59.578662 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:03:59.602478 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:03:59.612529 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:59.613886 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.618606 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.620882 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.623970 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.624539 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:03:59.627377 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 08:03:59.732509 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.741283 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.741809 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.749545 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.752136 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.757844 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.758864 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.770720 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.780912 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.791189 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.792405 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.792848 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.793546 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.796320 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.881728 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:03:59.886961 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:03:59.891026 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:03:59.891064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:03:59.895252 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:03:59.895645 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:03:59.903859 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:03:59.939989 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:03:59.999125 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.008996 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.019315 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.021657 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.044054 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.056586 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.060657 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.064337 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.068722 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.070231 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.077188 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.084247 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.097530 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.131069 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.139979 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:00.140211 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:04:00.142238 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.156603 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.158365 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.158555 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.159199 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.159859 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.171485 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.193744 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.194242 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.195057 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:00.206441 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:00.215365 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.216142 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.225132 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:00.265039 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.286087 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:04:00.326386 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.330797 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.333168 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.334467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.341446 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.355312 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.357273 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.423694 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.427453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.430132 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.445456 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.450469 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.452283 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.469928 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.471454 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.474270 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.474514 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.476037 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.484054 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.490062 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:00.492811 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:00.510910 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:00.514513 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:00.517433 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:00.575209 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.576357 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.579703 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.588402 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.591310 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.593889 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.629004 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.682020 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.683104 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:00.683933 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.685153 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.690783 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.693150 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.720022 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.771068 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.771334 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.772758 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.778292 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.780761 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.783146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.789465 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.791066 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:00.809746 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.809978 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.813808 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.825680 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:00.834954 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:00.835675 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:00.838258 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.865937 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:00.909672 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:00.913759 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:04:00.922986 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:04:00.950255 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:00.953547 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:00.953895 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:00.978899 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:00.995558 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.010529 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.011943 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.016517 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.029148 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.036567 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.045248 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.049228 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.057538 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.063925 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.075151 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.104440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.107220 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.107638 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.115254 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.133533 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.135623 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.145785 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.151659 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.154577 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.157472 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.159742 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.163633 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.184705 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:01.191928 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.223582 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.290681 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:01.311106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.311112 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.315851 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.323230 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.327121 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.332388 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:01.344241 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:01.356017 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.388079 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.393115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.412554 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.418586 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.430157 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.431008 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.435206 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.447793 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.452405 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.452641 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.462446 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.465427 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:01.466260 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:01.470165 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.472470 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.520258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.529527 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.540246 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.548106 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.551294 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.565222 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.566381 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:01.592672 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:01.636243 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.643502 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.645759 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.647444 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.669171 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.670384 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.670532 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.735321 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.738297 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.742791 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.747068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.748358 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.752630 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.753824 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:01.771443 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.775241 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.779067 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.798905 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.801556 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.810390 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.818070 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:01.844357 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 08:04:01.845892 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:01.889986 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:01.912221 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:01.921833 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:01.923957 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:01.926007 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:01.927289 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:01.929456 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:01.934934 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 08:04:01.951694 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:01.958370 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:02.017024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.022306 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.030737 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.035796 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.045994 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.051209 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.054486 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.066787 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.087625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.104110 12794 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.105664 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.111881 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.117455 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.130955 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.132267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.139299 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.143198 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.145402 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.147258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.148382 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.153661 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.158864 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.160740 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.174686 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.179440 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.194344 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.199730 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.207541 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:02.236006 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:02.248502 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:02.313673 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:02.333114 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:02.363512 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.376519 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.397615 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.399791 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.408293 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.411903 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.412976 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.427019 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.430649 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.434131 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.435063 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.438761 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.450505 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:02.470902 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:02.472177 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.474005 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.475528 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.495637 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.498737 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.503466 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.505586 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.535349 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.649082 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.649770 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.654733 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.656586 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:02.663255 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:02.664542 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.664816 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.668300 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.672163 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.679081 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.683020 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.698973 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:02.704306 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:02.711392 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.711783 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:02.719733 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:02.723425 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.724452 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.728886 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.738006 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:02.748073 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:02.758790 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.761261 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.772830 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.786391 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:02.819253 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:02.893606 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.897653 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.899029 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:02.913843 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.919386 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:02.932643 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.932953 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.933943 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:02.934585 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.948510 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:02.958287 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:02.970041 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.971514 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:02.984401 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.062987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.070557 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.071087 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.083863 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.085474 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.096961 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.100648 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.110401 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.116154 14638 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.116361 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:03.116381 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:03.138856 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.142406 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:03.149536 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:03.157939 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:03.164321 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:03.173612 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.220441 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.228550 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.230247 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.231138 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.232558 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.250046 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.255671 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.285005 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.291721 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.294627 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.299188 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.303308 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.314095 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:03.321441 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:03.332253 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:03.350309 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 08:04:03.412817 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.413067 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.421186 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.422042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.425825 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:03.427314 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:03.430888 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.464586 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.488941 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.505015 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.506337 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.509289 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.512015 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.532240 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.548079 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.572726 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.577744 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.578269 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.578299 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.595552 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.601249 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.614938 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.660903 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:03.677240 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:03.681088 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.681288 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.689643 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.695701 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.697526 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.700959 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:03.713706 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:03.739882 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.745116 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:03.763810 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.763991 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.764604 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.765645 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.766270 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.786584 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.788254 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:03.810070 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:03.839449 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.841158 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.846768 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.849164 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.849174 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.879238 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:03.881168 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.958564 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:03.964489 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:03.974459 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:03.975164 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:03.985003 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:03.990864 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:04.000144 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:04.007138 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.007581 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.011932 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.024952 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.025161 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.045342 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.046377 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.051055 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:04.091533 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:04.127278 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.129513 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.139489 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.145942 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.147467 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.151460 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.156159 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.160828 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.165417 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:04.177174 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:04.186700 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:04.187192 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.191960 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.197068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.209803 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:04.224103 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:04:04.239341 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.305968 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.308108 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.312044 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.312135 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:04.319468 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 08:04:04.319669 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.329715 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.329833 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.334133 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.341646 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:04.351645 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:04.353703 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.368021 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.369783 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.391382 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.393532 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.460304 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.465159 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.466042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.470299 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.489632 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.500193 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.503966 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:04.509460 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:04.573804 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.576355 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.588101 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.588799 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.591468 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.618768 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:04.626143 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:04.650564 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.660478 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.667937 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.670536 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.671133 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.674022 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:04.686152 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:04.687873 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.726331 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.726671 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.726994 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.727777 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:04.740622 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:04.750881 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.755636 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.783738 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.792028 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.810401 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:04.810621 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:04.826663 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.838883 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.848805 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:04.858435 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:04.859117 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:04.859400 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.872112 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.873767 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.875854 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:04.879189 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:04.886809 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.904060 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:04.915411 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:04.992774 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:05.006368 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.007326 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.026137 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.030050 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.042083 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.048910 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.055675 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.091615 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.093420 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.098378 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.100674 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.103061 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.103078 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.120157 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:05.183624 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:05.206071 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:05.206816 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:05.242542 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.248857 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.249511 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.262425 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.281358 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:05.283147 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:05.304613 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.306959 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.307003 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.318360 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.331930 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.332854 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.349963 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.359109 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.360348 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.370090 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.382053 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.387980 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.390142 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.402539 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.405766 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.412837 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.426436 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.432278 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.436966 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.460207 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.465646 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.471316 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.471314 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.537175 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.541851 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.543129 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.548373 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.548612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.557657 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.560601 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:05.593217 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:05.618476 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:05.650862 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:05.661554 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.670991 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.680032 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.681126 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:05.688155 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:05.694967 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.695132 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.706152 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.711714 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.726575 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.727864 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.728574 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:05.753552 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:05.765564 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.775370 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.780031 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.792063 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.793032 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.794047 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.819300 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.834722 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.848409 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.859133 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.923460 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.929697 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.931232 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:05.944670 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:05.956054 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.958937 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:05.963192 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.969594 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:05.971071 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:05.988072 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:05.993541 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.014991 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.021126 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:06.021860 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:06.025427 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:06.077495 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:06.119298 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.119560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.137554 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.140431 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.142405 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.143661 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.168866 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.174652 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.176230 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.183374 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.192780 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.194911 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.213323 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.227720 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:06.237394 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:06.238848 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.248847 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.250362 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.261021 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.262305 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:06.270653 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:06.282759 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.287361 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:06.317108 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:06.366357 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.381639 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.386648 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.388170 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.396485 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.397639 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.397900 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.444118 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.457975 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.460103 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.461393 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.494108 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:06.502869 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:06.503922 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.504362 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.543005 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.577687 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.578534 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.582374 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.587803 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.590605 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.605803 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.612205 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.613235 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:06.616851 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:06.620524 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.623769 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.637305 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.641402 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.662529 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.684482 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:06.690646 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:06.697355 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.703663 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.711937 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.717257 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.732611 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.735332 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.770888 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.773634 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.780462 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.784878 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.795418 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.799407 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:06.805121 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:06.809626 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:06.810132 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 08:04:06.865087 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:06.865972 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.869829 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:06.874682 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:06.877583 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:06.880892 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:06.900776 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:06.913345 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:04:06.914017 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:06.961714 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:07.008183 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.013451 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.019096 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.035441 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.037207 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.040223 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.064378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:07.111980 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:07.131058 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.132035 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.136938 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.138528 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.140372 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.172072 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.174413 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.207588 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.218143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.221060 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.225401 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.227174 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.237735 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.271315 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.279143 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.284999 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.286460 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.288218 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.296897 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.301908 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:07.310838 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:07.333235 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:07.333499 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.401707 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:07.407159 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.411196 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.415225 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.421155 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.421550 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.422308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.435094 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:07.535600 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:07.537014 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.539499 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.540872 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.544735 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.545023 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.548382 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.566053 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.567041 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.572841 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.577785 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.596410 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.597702 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.603037 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.604177 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:07.611744 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:07.623003 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.624693 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:07.629655 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:07.642586 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.657280 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.663600 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.677587 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.695319 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:07.701915 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:07.718670 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.719270 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.720832 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.721191 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.723076 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.746490 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.753260 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.814234 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.816838 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.817132 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:07.820128 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.840734 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.843160 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.844125 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.897385 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:07.908179 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:07.909804 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:07.910264 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:07.910857 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:07.920495 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:07.925385 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:07.928069 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:07.953228 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:07.983466 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.000340 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.006700 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.013679 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.031330 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.032356 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.085245 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.134591 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.137392 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:08.141664 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:08.142182 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.144896 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.151403 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.158433 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:08.162535 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:08.165570 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.169659 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.185093 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.187222 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.190042 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.191133 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.193202 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:08.211567 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:08.231724 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:08.241958 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.304474 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.306190 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.312114 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.312789 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.315733 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.331213 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.351776 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.382386 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.389403 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.389467 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.389703 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.395704 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.404644 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.427258 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.459123 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.473737 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.475447 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.489421 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.498976 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.499984 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:08.507786 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:08.517194 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:08.528323 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:08.547238 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.551489 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.555382 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.571323 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:08.573654 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:08.576277 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.580796 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.584357 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:08.592128 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:08.645632 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.653959 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.657752 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.664394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.675428 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.694808 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.698392 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.716158 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.723160 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.730299 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.730979 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.735054 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.737579 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.741323 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:08.748417 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.752727 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:08.813637 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.819876 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.825717 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.826265 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.829835 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.873729 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.874663 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.886605 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.892503 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.896658 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.904053 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.905440 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:08.921146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:08.922031 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.984740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:08.992992 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:08.997726 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:08.998299 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:08.999085 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.000218 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.039757 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:09.092212 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:09.096796 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.097519 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.098009 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.101066 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.102609 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.107928 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.119870 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.133752 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.136525 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.154753 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.155318 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.155717 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.165936 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.169270 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:09.218873 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:09.231693 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:04:09.244876 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.250205 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.253423 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.253791 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.256346 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.258194 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.259222 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:09.289520 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:09.300406 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:09.362396 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.362818 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.384001 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.385418 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.392884 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:09.395468 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:09.406169 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.411790 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.416643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.422237 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.430415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.440904 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.445681 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:09.447603 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:09.448295 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:09.468027 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.487583 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.500316 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.525740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.525997 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.537493 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.539044 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.540534 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.545125 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:09.549185 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:09.549911 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.551178 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.557513 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.564263 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.567101 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.571441 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.602973 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.687391 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.693404 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.694980 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.699781 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:09.700261 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:09.706344 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.709694 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.722477 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.753906 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.754774 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.755493 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.755789 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.763166 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.780668 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.799731 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:09.821457 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:09.923138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:09.927340 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.929356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.933379 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:09.954346 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:09.961802 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.970062 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.975061 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.982059 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:09.987696 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:09.999064 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:09.999322 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.003726 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.010461 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.015403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.028905 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.029997 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.053654 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.063977 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.066778 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.068348 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.074270 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:10.076000 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:10.083926 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.085538 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.106650 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.106829 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:10.131287 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:10.133591 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:10.136595 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:10.153241 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:10.219635 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.223459 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:10.282639 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.282696 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.292115 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:10.293311 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:10.293434 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.301051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.301280 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.306710 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.321768 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.322346 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.322482 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.343912 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.351503 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.365444 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:10.378423 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:10.379961 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.382448 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.398813 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.399570 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.400723 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.402571 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.412576 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.421195 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.484110 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.487114 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.488494 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.494254 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.495148 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:10.518083 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:10.528313 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.533689 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:10.555641 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:10.606228 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.607421 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.611399 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.615403 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.625349 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.633989 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.646900 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.654867 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.655473 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.666749 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.679358 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.679543 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.681605 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.684042 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:10.736585 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:10.765451 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.770259 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.774338 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.780525 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.792043 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.805804 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.848201 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.853034 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.859913 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.866786 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.878381 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.890590 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.903096 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.908022 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.908725 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:10.909828 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.936869 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.937705 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:10.937775 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:10.938707 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.943068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:10.953464 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:10.955019 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:10.972224 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:10.975432 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:10.982671 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:10.988509 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:10.988718 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.007342 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.011404 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.012693 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.112116 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.123309 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.123485 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.129112 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.142004 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.146198 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:11.156761 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:11.160877 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:11.161834 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:11.164510 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.169440 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.173048 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.175712 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.181468 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.185837 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.205875 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.226657 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.279917 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.284068 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.284307 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.300211 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.310588 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.318087 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.339442 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.375487 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.388911 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.391927 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.392457 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.395056 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.395905 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.396131 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:11.438601 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:11.462944 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.466115 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.469750 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.471339 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.493324 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.499184 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.502427 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.563848 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.564559 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.572991 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.591445 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.596233 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.606293 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.609849 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.610105 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.612828 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.624315 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.625931 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:11.632274 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:11.636407 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.647054 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.648352 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:11.683954 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:11.735994 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.737063 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.749511 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.752092 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.754346 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:11.759548 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:11.768965 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.788931 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.797387 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.798880 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:11.806097 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:11.818511 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.821851 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.834162 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.843446 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.852003 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.874647 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.883594 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:11.884001 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.886976 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.888480 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:11.899109 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:11.909035 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:11.920521 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.979899 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.980983 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:11.981225 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:11.982945 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:11.985975 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:11.992162 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.000195 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:12.048482 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:04:12.049207 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:12.095899 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.104607 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.105943 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.109509 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.110226 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.128376 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.129843 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.175022 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.188705 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.196038 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.197614 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.199033 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.202062 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.224336 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.227813 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.230422 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:12.235884 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:12.242158 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.244741 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.253199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.288678 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.295729 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.305473 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.312453 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.315672 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.321453 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.326869 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.328357 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.353292 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.454731 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.455874 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.459619 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.461654 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.467268 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.482784 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.483834 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:12.517381 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:12.521943 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:12.526254 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.532636 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.533592 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.534775 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.547008 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:12.549122 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:12.553462 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.561640 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.562671 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.572966 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.576434 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.581161 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.583145 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.607838 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.610584 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:12.642624 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:12.688228 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:12.690008 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.692807 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.702369 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.705067 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.719537 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.727098 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.741026 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.751135 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.754477 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.758445 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.758623 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:12.760279 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:12.763831 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.767071 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.784881 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.844403 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.847890 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:12.853880 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:12.856027 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:12.862974 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:12.872452 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:12.897620 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:12.899926 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:12.906687 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.974910 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:12.988449 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:12.999620 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.007314 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.009582 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.018204 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.027324 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.040444 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.046492 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.047721 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.050904 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.053036 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.053229 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.065738 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:13.104405 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:13.131596 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.137054 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.137272 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.137270 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.145671 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.146006 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.159955 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.181995 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.185526 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.187090 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.200862 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.204721 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:13.209702 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:13.211345 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.212041 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:13.217790 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:13.280050 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.281062 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.283371 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.291107 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:13.292011 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:13.294977 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.304641 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.315909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:13.333849 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:13.366783 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:13.425402 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.434010 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.441419 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.445484 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.447269 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.459175 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.473614 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.477139 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.484124 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.501034 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.512339 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.513521 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.520642 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.534410 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:13.576031 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:04:13.579844 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:13.580592 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.587117 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.592435 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.595278 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:13.597033 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:13.612603 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.615830 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.617336 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.620203 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.627157 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.633647 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.641448 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.648530 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.649365 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.649376 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:13.770138 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:13.778309 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.785512 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.789660 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:13.794330 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:13.805301 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.805958 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.806147 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.807271 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:13.812615 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.818357 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.819468 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.822659 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.834527 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.843530 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.856081 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.914008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.914956 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:13.918565 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:13.922286 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:13.945853 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:13.946336 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:13.948606 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:13.988056 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:14.074085 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.074395 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.082875 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.095297 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.105484 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.106787 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.123256 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.125803 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.126400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.128033 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.131086 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.143954 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:14.146211 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 08:04:14.148742 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.150844 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:14.154021 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:14.157140 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.160077 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.169101 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.169340 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.209656 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.215174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:14.219204 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:14.223563 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:14.227784 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.228879 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.233481 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.250756 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.255436 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.259246 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.272171 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.282581 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:14.315446 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:04:14.378427 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:14.395735 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.400204 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.409106 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.417727 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.419337 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.431231 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.461313 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.472282 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.475335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.477186 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.481787 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.484882 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.485205 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.487185 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.493418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.501852 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.508150 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.511615 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.525817 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.558825 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.567471 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:14.654633 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:14.706840 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.717415 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.722702 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.737080 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.740769 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.765471 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.768033 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.770720 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.781697 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.789490 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.791574 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.792073 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.793133 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.793332 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.808100 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.818955 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:14.820319 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:14.824193 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.825002 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.829797 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.829895 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.830174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.830183 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.834160 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.837548 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.847110 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:14.857235 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:14.867873 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:14.873243 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:14.883927 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:14.885805 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.892781 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.914680 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:14.921795 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.935252 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:14.937667 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:14.958860 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:14.976325 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:14.997328 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.067540 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:15.082670 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.082703 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.082954 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.090866 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:15.092453 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:15.096949 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.099360 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.110901 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:15.158157 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:15.199978 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.205214 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.206743 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.211596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.225227 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.228915 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.229449 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:15.269093 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:15.340369 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:15.343003 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:15.353916 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.356388 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:15.358749 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:15.373690 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.375167 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.383543 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.385814 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.392117 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.399176 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.417626 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.430411 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.436884 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.439425 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.446292 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.453249 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.457603 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.460085 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.468251 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.486425 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.491658 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.526356 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.572705 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.573940 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.576128 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.581872 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.585682 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.603189 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.614188 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:15.650708 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:15.673775 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.675601 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.680142 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.689394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.690685 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.701331 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.702980 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.757170 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.758872 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.758881 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.759618 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.771569 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.772908 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:15.776044 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:15.778239 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:15.806334 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:15.862274 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:15.877415 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.879906 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.881793 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:15.897294 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 08:04:15.899848 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.900141 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.941334 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.942625 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.947471 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.948951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:15.949846 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:15.950374 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:15.954700 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:15.958365 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:15.970266 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:15.994990 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.003278 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.004652 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.005666 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.016218 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.026461 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.033509 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.038147 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.038688 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.065819 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.069278 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:16.073581 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:16.088115 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.094254 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.105988 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.207350 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.208248 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.226393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.229350 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.232149 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.235069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.263626 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:16.295900 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:16.305389 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.306255 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.306286 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.310428 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.315479 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:16.323313 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:16.330708 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.343113 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.358441 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.374131 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.374701 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:16.377833 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:16.387924 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.409349 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.418394 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.419364 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.457072 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.459333 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.460139 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:16.461148 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:16.475884 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.477636 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:16.488252 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:16.504099 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.539336 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.565029 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.588505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.591228 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.592754 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.599164 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.635425 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.635557 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.647603 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.649109 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.659518 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.671996 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.673708 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.673947 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.677944 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.688127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.692642 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:16.698300 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:16.698558 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.699018 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.707347 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.714773 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:16.715724 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:16.725677 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:16.735072 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.798441 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.801478 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.813594 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.813850 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.819677 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.820008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.850960 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:16.888667 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:16.912402 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.913498 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.914541 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:16.915874 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.916810 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.923039 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:16.928225 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.930202 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:16.941155 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:16.948921 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:16.951063 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:16.963340 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.003134 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.013501 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:17.076144 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:04:17.077402 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:17.093274 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:17.103325 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.120617 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.120725 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.124909 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.125167 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.144578 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.151316 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.156859 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.157244 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.176467 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.180289 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.186722 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.205741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.208907 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:17.225126 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:17.266474 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.268589 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.274519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.291769 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.293390 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.303848 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.315572 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.362350 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.371235 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.373955 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.401099 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.411247 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.411470 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.413496 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.414470 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.416050 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.427203 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:17.431063 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:17.447866 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.456859 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.461261 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.469429 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.487145 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.493490 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.497385 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.499184 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.504757 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.509802 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.516171 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:17.540284 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:17.596957 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:17.629079 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.633890 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.654649 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.657125 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.672051 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.674634 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.675590 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.685838 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:17.685951 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:17.688992 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:17.695281 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:17.697445 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.700126 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.716589 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.718489 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.728427 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.735759 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.738230 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.739300 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.746713 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.763924 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.765868 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.797631 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.802122 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:17.834650 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:17.881997 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.884145 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:17.892815 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.893550 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.902873 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.906095 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.916106 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:17.917940 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:17.922688 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.941251 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.951187 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:17.953274 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:17.961021 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:17.962879 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:18.093039 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:18.113500 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:18.115684 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:18.123394 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.134829 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.135066 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.154362 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.155299 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.158666 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.162629 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.171454 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.190308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.201702 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.206763 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:18.208539 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:18.208858 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.213187 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.217500 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.218989 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.226732 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.237900 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.238600 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.244334 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.249387 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.281250 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.295583 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.313735 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.325007 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:18.339008 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:18.339479 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.352658 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.358709 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:18.372318 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:18.407569 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:18.436187 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.439357 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.448464 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.448905 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.456589 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.470685 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:18.480482 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:18.488014 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.518132 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.518870 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.523542 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.533563 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.539261 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.543886 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.553673 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:18.561086 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:18.597967 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.599665 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.603503 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.612334 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.613297 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.615381 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.623726 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:18.687870 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:04:18.690016 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.721601 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.729302 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:18.734274 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.739444 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.744153 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.746004 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.763060 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.776400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.788239 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.789882 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.809667 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.811504 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.814085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.821020 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.826326 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:18.829784 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.851759 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.858386 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.868650 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.872390 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.879124 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:18.884305 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.887543 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:18.944253 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:18.964687 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:18.966010 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:18.969918 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.979528 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:18.987457 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:18.997534 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.001982 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.019495 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.020648 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.030153 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.030354 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.032199 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.038524 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.059705 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.156217 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.157328 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.158195 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:19.172578 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 08:04:19.178748 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.190300 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.191726 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.195466 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.203939 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.208209 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:19.214226 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:19.219261 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.231177 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.232275 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:19.235198 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:19.252933 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.255677 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.344287 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.345850 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.355901 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.357807 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.371706 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.379354 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:19.386116 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:19.400328 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:19.421268 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:19.431118 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.441963 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.449291 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.459419 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.466331 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.467156 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.473158 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.501116 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.506086 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:19.508388 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 08:04:19.510208 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.521663 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.525084 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.532331 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.554152 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.594157 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.596024 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.597890 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.609951 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.617806 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.626210 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.635404 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.683774 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.688216 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.689358 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.695819 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.705039 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.714453 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.715263 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.717031 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.720101 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.730988 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.731578 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.746771 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.772958 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.779412 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:19.785959 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:19.786504 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:04:19.792482 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.796406 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:19.896179 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.896606 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.897440 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:19.909482 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:19.920468 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:19.921473 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:19.939560 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:19.961076 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.965437 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.974170 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.982092 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:19.988131 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:19.990195 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:19.991297 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.992064 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:19.994382 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:19.995499 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.005071 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.008311 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.011904 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.015947 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.017936 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.023109 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:20.030110 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:20.051520 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.062228 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:20.064714 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:20.065174 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.065182 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.104477 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.118544 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.119622 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.215910 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.220170 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.230402 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.231791 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.231928 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.244475 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.253988 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.264611 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.266616 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:20.269071 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:20.274113 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.301739 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.302139 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.317309 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.329896 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.340265 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:20.345455 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:20.345768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.355636 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.376147 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.388511 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.390944 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.394583 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.496937 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:20.500988 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:20.501546 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.512732 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.516799 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.532174 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.536649 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.541436 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.547365 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.556367 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.559911 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.564272 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.585498 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.594206 13057 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.594365 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.628305 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.632959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.633882 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.645051 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.649842 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.671972 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.679695 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:20.706033 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:20.712792 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:20.725384 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:20.755369 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.766541 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.767649 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.771076 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.786698 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.787366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.815250 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.832960 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.835772 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.846109 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.847365 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.853016 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.854843 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:20.872731 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.873104 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.873812 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:20.875553 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:20.883674 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:20.903091 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:20.919036 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:20.934254 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:20.981878 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:21.000115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.008626 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.010041 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.012871 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.013214 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.016013 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.030903 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:21.082106 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:21.105512 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.106200 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.114224 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.116505 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:21.122349 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:21.130722 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.144264 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.147976 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:21.168853 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:21.192198 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.196048 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.216681 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.219134 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.220870 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.221560 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.232172 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.267066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.268800 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.270581 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.279984 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.295954 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.301748 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:21.307499 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:21.326969 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.380935 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.386691 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.396484 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.398429 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.403345 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.406175 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.443948 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.454146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.458195 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.458412 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.459599 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.461598 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.477047 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.486068 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.567356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.575101 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.584046 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.585800 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.586262 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.586941 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:21.605187 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:21.622373 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.649796 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.654273 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.655025 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.656446 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.665611 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.665908 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.678429 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:21.678745 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:21.680140 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.681349 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.682114 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.690701 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.695906 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:21.702291 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:21.719074 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.731684 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:21.756402 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:04:21.757917 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:21.782783 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:21.799718 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.802727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.808966 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.810724 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.824785 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.831827 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.851505 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.902807 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.903625 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:21.907946 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.910550 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.915025 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:21.923445 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.925662 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:21.929877 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.931613 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.941715 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.943207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:21.951699 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:21.955354 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:21.967221 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.118111 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.130470 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.134654 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.141814 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.142472 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.156705 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.161252 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.161267 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.161504 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.175143 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.185801 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:22.187819 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:04:22.188558 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:22.191951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.197670 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.200644 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.227093 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.241022 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.278157 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:22.284382 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:22.286911 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.287012 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.291031 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.292484 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.300246 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:22.308023 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:22.332037 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.334165 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.341188 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.347108 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.364502 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:22.367295 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:22.376291 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.443362 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.445076 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.457038 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.457773 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.457957 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:22.482628 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:22.494143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.513116 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.531787 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.536913 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.541249 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.541517 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.547036 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.553174 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.571008 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:22.621783 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:22.626829 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.630754 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:22.632593 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:22.636224 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.640316 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.644325 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.656908 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.658679 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.671475 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.674687 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.676561 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.692883 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.702466 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:22.702466 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:22.706538 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.715837 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.763741 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.772784 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.774794 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.776983 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.791344 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:22.793888 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:22.800814 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.801393 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:22.808759 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:22.871742 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:22.871884 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.876747 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.879169 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:22.886538 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.888092 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.912329 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:22.932531 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:22.963351 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:22.968284 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.973915 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:22.977212 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:22.986454 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:22.986984 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.000223 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.000841 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.017833 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.030242 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.033105 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.037791 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.039299 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:23.051679 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:23.065017 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.176626 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:23.186105 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:23.190132 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.193222 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.193792 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.202396 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.203109 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.209599 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.216531 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.218238 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.219671 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.220614 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.245368 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.263041 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.267813 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.282250 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.287897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.301519 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.310324 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.311333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.316933 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:23.318984 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:23.354784 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:23.359871 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:04:23.367034 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:23.367765 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:23.408509 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.411656 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.416013 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.424414 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.428028 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.460618 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.478116 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.480343 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.482066 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.491611 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.507531 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.508872 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.510696 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.514758 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.606894 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.607157 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.610340 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.611047 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.612843 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.649509 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.655371 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:23.660624 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:23.680758 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:23.681941 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.685220 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.694505 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.698653 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.708616 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.719172 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.726006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.746238 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.752154 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.755038 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.760202 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.765929 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:23.776016 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:23.784924 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.790704 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.841254 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.842550 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.843243 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.843565 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.853034 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.862639 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.867837 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:23.871534 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.914553 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:23.933678 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:23.934002 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:23.936455 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:23.943697 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:23.946719 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:23.952215 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:23.975808 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.057296 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.066972 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.068476 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.069402 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.072845 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.088651 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:24.090728 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:24.092618 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.102419 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.105648 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.107884 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.116833 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.120227 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.143242 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.152968 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:24.178268 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:24.236881 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.247422 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.249862 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.257393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.259966 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.282325 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:24.305138 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 08:04:24.314976 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.325109 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.332912 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.346868 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.356222 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.359601 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.362289 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.362743 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.362957 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.372417 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.372884 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.389705 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.411698 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.412046 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.412640 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:24.469127 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.472438 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:24.479506 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.480429 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.484297 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.490742 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.499006 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.511931 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.513475 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:24.588975 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:04:24.595964 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.596765 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.600636 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.606274 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.616878 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.629778 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.631536 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:24.656559 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:24.664211 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.667109 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.668066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.677946 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.677973 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.678484 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.698287 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:24.726699 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:24.751670 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.757581 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.760111 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.771706 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.773842 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.779000 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:24.782727 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.793217 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:24.805346 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:24.860296 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:24.879873 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.889726 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.894500 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.901715 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.919366 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.920466 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:24.921468 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.930569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.930588 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.942286 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:24.953750 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.954545 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.957115 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.963449 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.980228 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.982705 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:24.984354 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:24.985044 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:24.993808 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:24.998648 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.050133 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:25.113459 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:25.116335 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.119007 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.120538 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.124191 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.124771 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.124982 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.145619 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:25.166143 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:25.167788 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:25.230633 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.231930 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.236279 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.237939 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.239341 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:25.242681 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:25.248823 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.248838 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.281327 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.284808 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.286085 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.287237 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.292279 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.292897 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.306878 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.391690 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.397464 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.399549 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:25.399870 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:25.409353 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.409565 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.426826 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:25.435824 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:25.438410 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:25.510550 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:25.511258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.521939 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.527325 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.540066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.540640 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.551404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.554119 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.555781 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.575801 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.579115 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.597486 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:25.612807 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:25.615392 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.618872 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.626878 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:25.635636 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:25.636796 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.639743 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.654395 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.657544 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.678416 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.685194 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.686902 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.690174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.691565 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.696470 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.710131 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.714475 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.726270 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.739053 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.837769 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.838076 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:25.840896 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.843731 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.853626 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.854065 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.861783 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:25.864384 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.881907 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:25.883189 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:25.890087 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.895907 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:25.905192 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:25.905472 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:25.936002 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:25.983822 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:26.032793 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.033316 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.046298 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.055238 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.061179 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.082113 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.082697 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.086392 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.097748 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.104782 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.107946 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.112273 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.127933 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.133184 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:26.134194 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.161217 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:26.220167 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.225430 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.228178 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.228804 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.230060 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.232496 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.246851 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.259747 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.267660 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.272029 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.275157 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.283617 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.286123 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.305032 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.312987 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.314314 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.333281 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.334659 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.339654 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.341858 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.347013 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:26.414222 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.451773 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:26.488546 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.495075 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.495618 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.498270 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.499096 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.509562 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.509979 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.527352 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.533607 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.534579 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:26.564373 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:26.576033 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.586081 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.586252 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.591395 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.596500 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.599867 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.607239 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.611819 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.629818 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.637631 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.637645 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.691744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.702204 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.703794 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.705842 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.710661 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.725250 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:26.737793 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:26.740106 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.771898 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.772145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.773602 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.776574 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.790517 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.796917 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.824724 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.857867 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:26.861915 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.868736 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:26.870165 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:26.875114 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:26.892153 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.900056 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.962766 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:26.973400 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 08:04:26.980486 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:26.989249 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:26.992548 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:27.000205 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:27.009871 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.023540 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:27.025799 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:27.034392 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:27.034801 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:27.067013 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.073305 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.080273 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.084897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.085172 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.088536 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.089097 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.169065 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.170481 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.175024 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.176440 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.177793 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.180758 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.183297 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.185907 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.201922 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.221390 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.221803 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:27.233733 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.235250 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:04:27.235756 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.235764 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:27.236514 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:27.264420 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.301666 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.302508 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.307505 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.310810 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.334589 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.335682 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.346329 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:27.362596 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.398206 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:27.444784 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:27.473186 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.477574 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:27.480288 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:27.482713 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.494518 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.503432 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.511268 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.541478 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.553972 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.565961 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.570101 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.587509 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.587731 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.587975 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.594017 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.594868 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.597172 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.623925 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.628702 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.646193 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.646677 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.649541 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.651181 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.659344 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.659579 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.659799 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.664911 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.696264 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.703246 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.728333 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:27.737306 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:27.738142 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.742813 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.743600 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.751432 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.753532 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.764561 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:27.814411 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:27.838083 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.841564 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.848088 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.850394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.860172 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.860814 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.862586 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.921499 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:27.923686 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:27.926098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:27.926944 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:27.929574 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:27.931596 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:27.942301 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:27.997478 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:28.078740 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:28.089051 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.096451 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.113417 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.113847 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.115063 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.117558 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.118023 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.120000 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.137540 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.152921 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:28.161652 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:28.163414 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.164837 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.164988 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.166436 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.178320 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.180085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.198652 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.201932 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:28.220299 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:28.223839 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.230295 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.235982 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:28.237670 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:28.287608 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.289148 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.296056 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.296203 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.296648 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.306488 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.313707 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:28.382174 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:04:28.383630 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:28.394932 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.398501 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.407241 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.411988 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.413378 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:28.417169 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:28.418360 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.423898 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.501319 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.506558 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.506976 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.512034 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.517699 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.525125 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:28.549482 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:28.560096 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.590204 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.590325 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.593600 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.596951 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.597956 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.618953 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.647207 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.663267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.675382 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.675475 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.682247 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.712821 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.723857 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.726449 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:28.759603 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:28.764465 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.766538 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.770109 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:28.774993 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:28.777200 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.785234 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.789119 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.800315 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.803566 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.811480 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.823662 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.827469 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.842411 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.853422 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:28.858875 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:28.907871 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.916146 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:28.951474 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:28.959920 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:28.961671 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:28.962654 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:28.983505 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:28.986109 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.004868 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.029963 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.031427 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.036154 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:29.041170 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:29.057255 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.059669 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:29.062165 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:29.064028 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.089028 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.184029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.186683 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.189888 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.192320 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:29.193356 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:29.195415 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.206892 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.208045 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.217753 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.247881 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.248109 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.248521 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.251358 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.252005 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.284016 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:29.310536 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:29.315754 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.315897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.318799 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.322126 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.333426 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.357784 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.375897 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:29.423621 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:29.431375 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.433482 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.435631 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.440814 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.444015 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.444916 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.456898 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.469532 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.470872 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.472921 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.488579 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.497231 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.503099 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.511394 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.566094 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.573441 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.581581 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.591974 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.592936 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.610505 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.610543 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.661197 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.664364 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.666075 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:29.670526 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:29.679108 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:29.683108 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:29.683964 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:29.698372 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:04:29.705839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.713948 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.732926 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.740391 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.744006 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.745489 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:29.753209 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:29.781034 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:29.787415 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.788782 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.796548 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:29.805892 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.808784 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.810367 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.812914 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.821717 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:29.839586 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:29.856545 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:29.952527 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:29.962656 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:29.972853 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:29.977304 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:29.982823 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:29.986712 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:30.008505 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.012173 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.013422 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.013859 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.015244 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.041826 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.045676 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.060233 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.068135 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.079835 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.080354 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.081553 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.096746 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:30.099576 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:30.100603 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.106691 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.133112 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.144580 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.154196 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.161885 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.167572 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.170401 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.176156 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.178802 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.186360 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:30.189473 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:30.204118 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:30.225610 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:30.321480 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.331096 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.331352 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.336639 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.340467 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.341492 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.393239 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.393637 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.399379 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.413434 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.415666 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.415903 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.429097 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.429483 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.441607 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.442842 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.447415 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.451200 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.463794 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.468775 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:30.478771 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:30.482527 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.486698 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.487375 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.488348 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:30.506278 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:30.527175 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.530593 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.530838 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.537942 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:30.554358 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.583532 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:30.627350 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.627488 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.632370 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.634078 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.639686 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.657105 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:30.658830 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:30.668171 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.701122 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.703613 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.712272 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.716356 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.716403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.720249 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.721614 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.791610 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:30.792848 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.793002 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.803071 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.803426 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:30.824245 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:30.829402 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.837674 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.882015 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.883150 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:30.888298 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:30.893256 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:30.898542 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:30.915367 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:30.921792 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:30.928701 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.970185 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:30.991308 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.003171 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.003822 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.019137 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.024441 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.024864 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.049813 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.057281 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.059588 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.067798 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:31.078290 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:31.090138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.091884 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.107044 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.108314 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:31.133546 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.168819 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:31.182809 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.193502 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.194011 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.202885 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.206538 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.227041 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.242938 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.255878 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.259534 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.283556 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.284516 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.290666 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.293532 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.295181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.299441 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:31.328284 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:31.334868 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.337625 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.338816 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.339071 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.343333 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.370263 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.426345 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.446189 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.449561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.452587 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.452684 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.465413 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.473575 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.537279 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.548991 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:31.550650 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:31.554618 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.559567 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.560029 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.569494 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.581552 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.614821 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.626345 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.635860 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.637495 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.663702 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.679519 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.680852 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.708921 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.713394 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.719244 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:31.724797 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:31.729610 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.738046 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.739720 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.739871 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.752146 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.754951 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.755241 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:31.771847 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.774471 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.775833 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.794517 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.890024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.894752 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.896914 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:31.900691 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.910530 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:31.913470 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:31.921556 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 08:04:31.950337 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:31.977850 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:31.997519 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:31.997968 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.010181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.013222 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.013836 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.017316 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.019068 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.023954 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.043648 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.046773 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.051235 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:32.073369 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:32.078220 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.095726 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.132393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.132880 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.132974 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.143071 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.143932 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:32.147265 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:32.155138 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.156448 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:32.178583 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:32.181303 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:32.217139 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.219522 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.228713 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.229743 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.229822 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.253664 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.259279 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.348131 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.353384 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.355823 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:32.357161 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:32.367867 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:32.368021 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:32.368606 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.370008 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.376710 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.389719 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.400568 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.400754 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.412887 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:32.414376 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:32.420219 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.420259 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.447108 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:32.487352 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:04:32.489187 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:32.494961 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.498147 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:32.505323 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:32.514534 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.517335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.520066 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.533648 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.534128 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.575757 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.578127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.581496 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.583184 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.583302 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.609583 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.623684 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:32.645871 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:32.669212 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.673385 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.682363 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.684720 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.691136 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.691886 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.709040 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:32.736429 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:32.782513 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.792235 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.794590 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.807066 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.808511 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.809260 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.816639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.817215 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.820418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.825352 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.861048 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.874097 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.893997 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.897742 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.897878 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.901890 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:32.911036 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:32.936122 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:32.939188 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.939327 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.945741 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:32.950384 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:32.954107 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:32.954506 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:32.954934 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:32.962638 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.983566 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:32.986343 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:32.993875 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.000075 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:33.035082 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:33.060240 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:33.101099 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.102578 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.102690 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.113924 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.114179 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.115562 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.138411 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.155587 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.160909 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.165356 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.166657 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.180764 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.192324 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.217099 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:33.271131 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:33.271754 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.278630 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.279631 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.279625 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.286479 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.287580 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:33.305831 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:33.318111 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:33.336773 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:33.386193 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.391023 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.393534 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.403412 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.404042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.412591 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.423628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.456292 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.463862 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.464632 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.474676 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.476042 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:33.483763 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:33.485883 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.488569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:33.534098 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:33.600322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.606495 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.613953 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.627755 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.640555 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.642693 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.659775 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.677892 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.678231 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.691380 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.695181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.701169 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.707414 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.731856 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.737661 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:33.738328 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:33.744879 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.745431 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.745746 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.755702 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.759047 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.762005 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.766254 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.772028 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.784507 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.794252 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.806351 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.813771 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.815232 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:33.861160 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:04:33.865885 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.919422 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:33.921767 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.924991 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.925380 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:33.939682 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:33.941004 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:33.943689 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:33.961212 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.961388 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.980978 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:33.982535 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:33.982829 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:33.987111 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:33.993717 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.003815 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.009073 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:34.046721 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:34.080806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.085369 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.089228 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.089485 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.089890 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.090039 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.094344 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:34.200381 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:34.204011 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.209203 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.210402 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.222211 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.231992 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.232800 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.234021 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:34.303179 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:34.305697 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.307729 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.308581 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:34.310835 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:34.312875 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.326141 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.333953 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.343088 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.344075 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.345654 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.358822 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.359586 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:34.361701 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 08:04:34.366693 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.405666 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.417500 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.433576 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.434207 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.435328 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:34.447176 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:34.448241 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.475312 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.477697 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.492264 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.495215 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.499778 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.522964 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.526170 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.528425 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.553007 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.553470 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.593899 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.596351 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.609432 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.609627 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.610291 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.631592 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.634297 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:34.679472 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:34.680716 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.682016 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:34.695712 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:34.701277 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.701328 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:34.707752 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:04:34.707824 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.720132 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.723400 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.761314 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.763602 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.780162 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.780256 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.791272 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.793272 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:34.829850 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:34.833614 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.839111 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.842957 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:34.844424 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.851347 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:34.852707 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:34.857205 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:34.890570 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:34.891222 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.914803 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.935614 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:34.940145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:34.949201 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:34.960274 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.993953 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:34.994961 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:34.996282 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.004511 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.013823 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.025962 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.027447 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.043447 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.053108 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:35.081779 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:35.095752 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:35.113876 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:35.119740 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:35.128525 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.145287 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.149823 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.154385 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.162221 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.163491 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:35.167492 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:35.172565 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.193876 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.211755 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.220075 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.226227 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.236147 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.239960 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.271252 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.339164 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.341361 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.342078 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.348361 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.349536 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.359077 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.390249 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.391068 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.399721 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:35.403393 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:35.410526 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.410786 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.426585 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.433588 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.456670 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.515205 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.518078 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.527259 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.530248 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.531090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.531507 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:35.555562 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:35.562950 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.595571 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.600733 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.607666 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.607949 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.610318 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.621143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.631300 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.659554 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.659863 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.671069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.671363 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.691749 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.706686 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.707592 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:35.715860 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:35.716382 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:35.718910 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:35.736495 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.736826 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.742501 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.747717 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:35.759478 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:35.769327 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:35.770828 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.771646 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:35.809768 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:35.844806 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.850093 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.850483 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.851059 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.858579 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.871490 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.873020 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:35.924858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:35.927306 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:35.929741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:35.929811 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:35.930526 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.960944 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:35.967922 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:35.968094 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:36.053771 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.069245 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.071666 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.080631 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.085152 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:36.088848 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:36.097247 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.099332 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.158119 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.159080 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.163672 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.169308 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.171012 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:36.176091 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:36.177209 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.184705 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.193645 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.198539 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.208061 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.208616 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.233342 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:36.247288 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:36.249495 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.257166 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:36.300513 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:36.310571 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.324537 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.324594 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.339277 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.347237 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.369793 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.375236 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.409073 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.416896 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.420436 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.420534 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.421437 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.427160 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.454967 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.473491 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:36.476455 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:36.479941 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.488236 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.488595 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.497424 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.501542 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.511999 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.523867 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.534212 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.542800 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.546772 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.558676 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.568316 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.572489 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.668192 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.668951 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.670185 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.671841 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.678580 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:36.710453 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:36.711470 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.713850 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.732781 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.733114 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.736270 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.737561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.742798 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.755378 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.770699 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.852330 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.858659 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.865401 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.882602 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.883065 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:36.883695 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.901171 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:36.918598 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:36.919555 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.930298 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.938303 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:36.941027 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 08:04:36.941394 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.946038 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.947085 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:36.951354 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.973670 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:36.976083 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:36.984714 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:36.986382 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:36.994009 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.046772 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:37.049469 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:37.078722 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.088809 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.089164 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.096127 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.109994 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.115628 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.122192 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.127059 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:37.181877 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:37.184609 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:37.215134 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.228596 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.238065 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.238056 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.239040 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.251705 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.259040 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.267741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.272377 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.282243 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:37.292244 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:37.304016 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.311596 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:37.328089 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:37.335769 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:37.336763 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:37.344986 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.345846 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.346319 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.365324 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.367887 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.369717 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.371055 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:37.386452 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:37.418090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.439131 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.442404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.442435 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.446946 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.460165 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.460839 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.464480 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:37.541067 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:04:37.541775 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:37.545820 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.547794 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.550253 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.561048 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.571173 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.579807 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.603021 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.604291 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.610459 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.614305 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.623428 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.631923 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.635069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.640096 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:37.643959 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:37.741573 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.745754 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:37.745793 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:37.746345 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.754271 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.763609 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.773361 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.793145 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.793743 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.799602 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.809834 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.821909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.822553 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:37.828333 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:37.840128 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.847687 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.879702 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.879985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.898591 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:37.900923 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.902392 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.907337 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.934051 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:37.939460 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:37.979108 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:37.980684 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:37.981437 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:37.982254 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:37.990186 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:37.993058 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.007630 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.128029 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.131466 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.135704 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.140332 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.143060 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.143080 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:38.151523 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:38.152115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.162299 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.166337 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.167768 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.188292 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.188652 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.198001 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:38.215016 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:38.215597 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:38.232031 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:38.253286 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:38.258767 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.269658 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.274371 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.281985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.282979 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.284999 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.303436 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.339727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:38.342387 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:38.342761 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.358119 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.361452 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.374082 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.374511 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.375820 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.425681 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.426031 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.426575 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.428844 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.429076 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.436718 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.445945 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:38.530643 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.562337 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:38.569581 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.571779 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.572017 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.576751 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.590531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.597489 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.600374 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.601615 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.608651 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.617760 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.624588 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.633244 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.638206 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.645395 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:38.745810 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:38.761222 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.764605 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:38.766510 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:38.772747 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:38.794024 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.795650 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.797255 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.797840 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.808702 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.809383 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.813125 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.820566 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.852869 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.856053 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.859094 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.866518 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.866797 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.881006 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.881476 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.898841 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.904045 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.907455 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:38.919328 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.923173 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:38.947269 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.949889 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:38.950814 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:38.953058 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:38.960165 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:38.989728 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:39.047976 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:39.097772 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:39.124825 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.128566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.134606 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.142696 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.144460 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.148010 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.151463 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:39.156174 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:39.173166 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.175554 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.176877 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.194482 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.201655 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.212181 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.223522 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.233736 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.239243 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:39.245604 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:39.248070 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.252449 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.262324 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.264357 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.286823 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.314157 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.323763 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.351065 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.351913 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.352753 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.354156 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.357726 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:39.448112 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:39.451535 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:39.451882 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 08:04:39.452811 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.452884 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.458701 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.458701 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.459199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.462579 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.553155 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.558604 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.559162 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.560406 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.564328 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.567653 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.571368 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:39.579624 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:39.591085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.598773 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.598950 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:39.604130 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:39.623194 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.624966 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.627928 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.656255 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:39.694953 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:39.714527 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:39.733976 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:39.736860 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 08:04:39.737568 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:39.742357 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.744124 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.745203 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.751065 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.753727 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.763034 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.785358 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.787518 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.793535 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.796777 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.806066 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.812948 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:39.841640 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:39.846050 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.893378 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.896751 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.900132 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.900393 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:39.908267 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:39.911219 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.934631 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.962388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:39.971704 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:39.991014 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:39.997576 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:39.997831 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.006428 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.025722 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:40.031652 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:40.080657 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:40.091127 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.093258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.096231 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.098596 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.101426 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.101522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.117491 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.132298 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.133236 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.140118 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.154033 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.160568 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.171473 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.184512 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:40.275992 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:40.279237 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:40.287531 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:40.305143 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.311612 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.315512 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.318944 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.326153 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.334635 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.334836 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.349985 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.357175 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.357496 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.366456 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.387389 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.412001 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.416441 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.449302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.449546 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.460958 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.467065 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.479723 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.487720 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.487993 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.489318 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.489733 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.512655 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.515908 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.521219 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.524421 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.535110 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.606454 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.607931 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.618613 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.620985 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.621752 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:40.624485 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:40.630995 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:40.636799 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.666756 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.700219 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.705945 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.706743 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.706784 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.711886 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.712735 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:40.733376 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:40.734589 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:40.752609 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:40.785271 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.788111 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.789881 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.800776 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.805385 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.805979 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.806474 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:40.839020 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.849128 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:40.884441 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:40.895573 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.899232 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:40.908646 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.912926 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:40.915973 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:40.932847 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:40.936584 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:40.988684 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:40.996237 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.001956 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.014132 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.038267 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:41.039090 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:41.043509 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.071681 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.082582 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.089031 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.094520 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.114032 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.126618 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.128125 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.129570 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:41.131750 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:41.133234 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.134516 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.157745 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.162488 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.170125 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.178546 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.184962 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.189577 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.190515 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.191568 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.193181 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:41.201435 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:41.209053 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.214342 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.219653 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.329973 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.330912 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.337594 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.338887 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.350431 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.357448 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.361028 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.396132 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.396338 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.398339 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.401355 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.408023 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.411556 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.417268 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:41.446137 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:41.539784 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.555341 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.560290 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.562590 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.581496 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.581578 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.602380 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.608584 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.640276 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.640960 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.643882 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.656934 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:41.669449 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:41.670106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.670796 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.685647 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.690258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.699843 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.701189 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.709359 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.715240 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.718650 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.724782 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.736057 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.752859 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.771625 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.774210 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.784343 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.795982 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.802174 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.804442 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.804783 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.817001 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.825759 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.852144 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.857352 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:41.881968 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:41.951068 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:41.951155 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:41.957644 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:41.958344 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:41.958447 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:41.964540 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:41.992769 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:42.007803 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 08:04:42.086901 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.090691 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.093158 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.093462 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.097108 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.114683 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.158670 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.163970 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:42.173439 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:42.174597 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.182873 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.187937 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.191366 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.198194 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:42.200757 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:42.209625 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.240058 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.248929 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:42.249812 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:42.250016 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.257839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:42.261420 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:42.274761 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.287869 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.312772 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.345561 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.345815 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:42.349105 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:42.362181 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.364461 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:42.397459 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:42.400055 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.401062 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.409250 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.432137 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.432174 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.434448 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.435861 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.454371 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:42.454602 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:42.465341 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.478827 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.544209 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.554594 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.557632 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.559549 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.562420 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.563580 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.566624 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.568370 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.571780 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:42.585438 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:04:42.617288 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.621855 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.628325 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.639549 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.641197 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.707190 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.709589 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.711637 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:42.718741 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:42.718761 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.723865 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.726542 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.742760 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:42.773999 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:42.785801 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.795269 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.798084 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.799865 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.801867 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.808985 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:42.817425 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:42.830118 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.875705 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:42.883211 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:42.884495 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:42.886726 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:42.888123 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:42.890336 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.899338 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.928047 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.972312 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:42.977625 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:42.987871 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:42.989185 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.000844 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.010947 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.031654 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.075600 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.080768 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.083094 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:43.085702 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:43.093869 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.100814 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.101531 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.111786 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.123674 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.124608 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.139356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.139739 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.154106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.162909 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.173959 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:43.191983 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:43.197680 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:43.211457 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.211839 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.216449 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.232597 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.235347 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.236855 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.243420 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:43.260995 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:43.317957 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:43.333561 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.337649 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.343796 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.348248 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.353114 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:43.355929 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:43.371320 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.385751 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.464581 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.467561 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.468735 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.475154 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.477641 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:43.487094 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:43.508242 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.510418 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.523823 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.526384 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.537012 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.544924 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.545385 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.546265 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.552537 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.561239 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.562575 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.563954 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.575588 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.586103 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.594677 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.594713 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.595144 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.599395 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:43.608467 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:43.610157 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.612479 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.617902 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.621590 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.625948 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:43.671577 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:43.747815 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:43.762493 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.767724 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.769369 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.780314 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.782848 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.794416 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.800480 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.820909 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.824561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:43.844622 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.845460 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.846643 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:43.848913 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:43.858834 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:43.968648 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:43.991300 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:43.993371 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:43.995771 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:43.995887 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.004612 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.018049 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.043226 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.046046 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.052735 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.056790 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.061203 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.063642 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.066457 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.074422 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.077107 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.110975 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.113907 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.145735 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.149538 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.158811 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.158809 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.159997 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.165016 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.168200 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:44.181587 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:44.182965 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:44.194607 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:44.217047 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.225916 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.227979 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.316318 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.320546 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.337664 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:44.351205 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:04:44.352892 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:44.368587 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:04:44.389284 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.390290 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.391463 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.395532 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.402949 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.404140 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.421651 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.428966 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.433296 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.435626 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.447361 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.447675 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.448133 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.462985 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.463778 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.465665 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:44.468115 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:44.484175 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:44.517529 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 08:04:44.596654 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.597076 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.599493 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.600447 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.609300 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.609477 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.656605 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.657627 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.665760 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.667815 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.676759 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.677955 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.687296 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.688500 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.707237 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.710091 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.721263 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.722879 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:44.728600 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:44.734870 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:44.736505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.738839 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.740370 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:44.749938 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.762708 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:04:44.770049 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.795352 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 08:04:44.850260 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.861773 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.864293 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.871606 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.881971 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.886415 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.903621 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.931650 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:44.934501 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.942902 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.960700 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.973198 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:44.973323 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:44.983381 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:44.984117 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:44.991452 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:44.996243 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.002068 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.022527 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:45.041152 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:45.054738 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.057325 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.063499 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.066043 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:45.069316 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:45.080204 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.083192 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.093111 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.101269 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.110209 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:45.224025 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:45.245450 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.263952 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.264667 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.277920 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.290122 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.293044 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.294687 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.296257 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.299475 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.309646 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.310293 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.341513 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.347975 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.355477 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.356423 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.364598 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:45.368000 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:45.368659 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:45.386504 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.387169 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.387966 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.399547 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.403055 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.434146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.438226 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.446331 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.448029 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.454377 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.463932 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.477880 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.486059 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.497807 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.499853 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.504103 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.507687 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.509315 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.511130 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:45.605759 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:45.640121 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.642109 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.643013 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.649856 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.662560 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.671989 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.672276 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.683624 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.687322 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.703106 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.710103 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:45.715924 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.716996 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:45.719267 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.719377 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.744177 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:45.811837 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:45.818274 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:45.825985 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:45.834154 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:45.837954 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:45.845265 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:45.866175 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:45.884106 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:45.898833 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.904735 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.910614 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.922175 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.925519 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:45.927824 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.951836 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:45.953214 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.956030 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:45.961889 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:45.967732 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:45.999229 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.001518 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.004043 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.006395 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.017799 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.019991 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.020241 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.023406 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.035909 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:46.040961 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:04:46.053963 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.057327 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.070950 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.078727 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.138686 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.139537 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.141458 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.141791 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.146211 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.148213 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.150787 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.250716 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.259557 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.264194 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.264616 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.267858 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.269600 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.273139 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.274708 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.285399 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.307260 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.320219 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.320472 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.321919 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.327370 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:46.361434 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:46.393842 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:46.396277 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:46.423964 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:46.452144 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.454959 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.459568 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.462880 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.473649 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.474074 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.485468 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.495515 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.503244 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.504662 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.507091 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.508081 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.509434 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.514230 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.594350 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.595439 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.597353 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.607213 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.612787 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.620628 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.625973 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.710289 12926 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.710814 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.711447 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.711917 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.719919 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.734400 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.735491 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.745196 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.752926 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.759688 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.761190 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.761324 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.761694 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:46.765599 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:46.806531 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:46.846582 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.849969 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.856417 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.860920 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:46.876502 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 08:04:46.877552 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.878067 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.888161 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:46.892105 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:46.946323 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:46.946951 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:46.962141 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:46.962762 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:46.973928 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:46.980546 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:46.980727 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:46.980953 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.017897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.035912 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.037066 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.059939 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.060506 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.062450 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.093554 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.135653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.137177 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.142668 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.142783 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.166244 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.167098 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.168228 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.172480 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.173086 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.180262 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.193255 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.196192 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.219199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.226622 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.247363 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.252830 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.253465 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.270085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.277745 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.281881 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.299059 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:47.306612 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:47.346331 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:47.366200 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:47.407281 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.409950 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.412392 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.416697 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:47.443667 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:47.444710 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.450809 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.452826 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.454545 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.477563 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.478792 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.479983 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.484046 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.486966 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.498374 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:47.502990 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:47.511205 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:47.527513 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.530678 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.535382 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.538679 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.551630 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:47.576797 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:47.581157 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.590406 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.600567 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.602680 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.602823 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.636732 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.638573 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.638813 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.661355 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.725589 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.735145 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.741202 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.743551 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.747877 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.755183 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.759930 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:47.798771 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:04:47.799628 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:47.815322 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.816126 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.819909 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.820158 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.827114 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:47.830428 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:47.831266 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:47.861292 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.871268 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.886082 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:47.888473 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.900324 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.904467 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.917075 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:47.921805 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:47.921905 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:47.937323 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.938786 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:47.944834 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:47.948192 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.951162 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.961606 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:47.961752 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:47.986265 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:48.034070 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:48.058436 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.058486 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.076803 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.089805 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.094718 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:48.099599 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:48.107028 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.130331 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.187209 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.188618 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.190901 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.212903 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.218029 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.224187 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.231503 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.233430 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.236086 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.239238 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.240952 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.243198 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.263576 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.265336 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.286213 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:48.290307 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:48.291565 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.294140 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.297328 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:48.303084 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:48.309867 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.319173 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.328979 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:48.416015 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:48.473403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.488399 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.489955 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.498319 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.507723 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.510514 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.510529 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.512539 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.512540 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.527011 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.546377 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.547709 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.556435 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.592067 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.599438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.602751 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.612716 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:48.616575 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:48.621197 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.632484 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.641867 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.643743 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.645213 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.646121 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.648699 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.656585 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.700158 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.703606 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.730834 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.774616 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.774878 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.777814 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.780489 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.787586 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.791113 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.814512 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:48.818084 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:48.825628 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:48.862375 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.865857 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.867522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:48.867604 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:48.876026 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:48.887902 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.912685 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:48.966341 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:48.967643 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:48.975064 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:48.975421 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.985531 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:48.991473 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.025097 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.029888 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:49.060282 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:49.062789 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.069581 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.071120 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.076846 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.086221 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.097492 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.114360 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.163317 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.167971 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.168215 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.172250 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:49.175546 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 08:04:49.180508 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.183250 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.200069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:49.254859 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:49.276819 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.280180 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.286509 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.326040 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.331310 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.344815 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.359179 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.363601 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.368100 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.369104 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:49.372610 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:49.394634 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:49.394658 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.404069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.404839 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.438604 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.438894 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.447502 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.448042 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.459359 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.462389 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.477067 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.480311 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.483265 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.486219 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.487587 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.499750 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.499804 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.502067 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.555069 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:49.567986 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:04:49.598743 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:49.625874 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.636492 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.640259 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.642669 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.646364 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.664574 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.670459 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:49.672339 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:49.726228 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.726228 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.739562 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.742973 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.751773 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.755223 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.764416 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.776670 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.780692 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.784161 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:49.787775 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:49.799646 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.803647 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:49.833406 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:49.836484 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.840731 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:49.858896 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.861052 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.866861 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.867133 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.867868 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.871891 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.895853 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:49.913767 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:04:49.922905 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:49.954514 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:49.964406 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:49.965148 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:49.975497 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:49.978152 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:49.994555 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.000064 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:50.050308 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:50.097079 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.099392 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.105722 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.110818 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.112147 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.119750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.132335 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:50.140285 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:50.142454 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.154906 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.169098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.172510 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.173609 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.179679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.180392 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:50.185010 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:50.189656 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.194932 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:50.197104 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:50.198132 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.215348 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.216051 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.219395 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.228943 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:50.259676 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.336092 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:50.357193 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:50.395565 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.396504 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.396605 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.400733 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.404215 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.418457 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:50.424554 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:50.430444 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:50.439559 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:50.443725 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.448763 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.452488 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.459532 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.460862 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.461069 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.463759 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.476480 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.477236 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.483063 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.484988 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.488523 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.507532 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.509466 14769 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.510197 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.523501 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.535013 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.552281 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.556807 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.557669 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.559439 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:50.602494 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.647532 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:50.661891 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.668974 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.679095 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.683677 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.683916 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.700783 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.704999 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.767148 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.767199 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.772100 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.777287 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.779186 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:50.782060 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:50.787846 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.815538 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.821170 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.823153 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.823819 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.823849 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.824443 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.827580 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.853381 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.893617 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:50.895825 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:50.899448 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:50.899577 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:50.920897 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:50.926847 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:50.926873 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.003939 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.006292 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.019002 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.019841 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.032228 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.032850 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.052811 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.066972 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.070322 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.095434 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.102452 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.106140 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.111092 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.114277 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:51.152655 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:51.170866 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.171427 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.171968 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.172921 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.183972 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:51.186349 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:04:51.203792 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:51.212745 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.213478 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.273095 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.278340 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.282569 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.283780 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.291224 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.294695 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.315425 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.384905 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.390208 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.390460 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.399044 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.399830 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.400681 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.400898 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:51.402868 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:51.441705 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:51.479938 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.484025 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.493242 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.495249 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.496981 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.505863 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.508781 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.561494 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.569478 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.575206 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.577482 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.608111 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.613281 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.616060 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.616753 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.620311 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:51.631321 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 08:04:51.634439 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.636456 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:51.648552 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:51.652870 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.655723 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:51.668062 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:51.668484 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.730854 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.735152 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.743845 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.747145 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.747761 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.748831 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.758296 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.761765 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.787225 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.788156 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.791591 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.799288 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.801980 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:51.805655 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.890504 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.893761 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:51.897059 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:51.899089 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.904335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:51.913305 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.913952 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:51.931934 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.971206 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:51.975303 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:51.981696 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:51.991479 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.002683 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.017098 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.020846 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.021117 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.025935 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.029181 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.031837 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.051880 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.070693 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.078533 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:52.150266 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:52.177137 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.188752 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.190224 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.193444 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.193890 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.198984 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.205583 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.226611 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.227141 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.239617 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:52.244752 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:52.250275 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.263155 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.268716 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.273161 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.316560 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.327394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.331435 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.332473 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.333514 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.345974 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.373636 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.400631 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.406838 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.412138 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.426980 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.429744 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.430339 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.447405 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:52.469305 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:52.528669 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.531420 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.535072 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.538506 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.557276 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.564335 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.573470 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.578024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.582937 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.584594 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.587617 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.627067 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.627370 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.645992 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.648620 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.650192 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.657521 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.683380 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:52.686226 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:52.686646 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:52.692453 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:52.698442 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.700464 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:52.705504 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.741264 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.755430 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:52.756270 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.758033 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.760233 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.763288 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.767204 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.774863 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.775367 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.834859 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:52.841825 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:52.842077 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.844745 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.845727 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.849593 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.866657 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.868280 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:52.886626 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:52.887576 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.905167 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:04:52.905933 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:52.926214 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:52.945595 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.954636 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.961759 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:52.966964 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:52.969619 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:52.976326 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:52.990880 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:52.994185 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:52.994976 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.002851 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.003118 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.011559 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.020442 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.020731 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:53.048275 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:53.116901 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.120256 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.123313 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.125499 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.126545 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.137504 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.157842 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.182883 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.184687 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.186906 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.191177 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.201514 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.215273 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.228457 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.280791 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.285506 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.289419 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:53.291134 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:53.293782 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.313318 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.316830 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.323263 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:53.370615 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:53.385969 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:53.401729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.403949 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.408936 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.424998 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.428176 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.428887 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.440814 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.466288 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.475621 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.477530 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.479431 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.485455 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.491511 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.492920 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.568490 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.570346 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.574084 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.591657 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.593547 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.601908 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.605139 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.606346 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.612720 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.613351 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.623162 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.627430 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.631788 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:53.659868 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:53.672482 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.699775 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.700026 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.701622 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.705289 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.706073 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.732851 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.742043 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:53.751293 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:53.756143 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:53.809978 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.811717 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.815272 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.818564 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.833554 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.841907 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.843098 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.905886 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.913100 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.913710 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:53.914817 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:53.928807 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:53.942299 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.942989 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:53.949105 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:53.964430 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.964716 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:53.967398 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.968755 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:53.983654 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:53.993383 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.021875 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.076692 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.100065 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.108541 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.109934 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.111526 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.116437 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.126942 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:54.129238 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:54.150347 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.165287 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.172518 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.175127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.177165 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.199440 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:54.207801 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 08:04:54.211406 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.236912 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.246912 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.257663 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.258349 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.258592 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.271066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.277153 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:54.351817 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:54.360045 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.366147 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.366748 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.367518 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.374750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.402005 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.411857 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.415995 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.419301 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.445482 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.449877 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.451503 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.457162 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.467132 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.502681 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.506981 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.513742 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.514840 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.518072 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.527416 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:54.539803 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:54.541411 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.559520 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.560251 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.568918 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.572427 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.587270 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.602418 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.603905 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:54.679168 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:54.683609 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.684486 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.703019 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.711108 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.711412 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.721292 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.722185 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.731498 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.745332 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.750473 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.752264 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.759212 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.759639 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.771939 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:54.794601 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:54.822355 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:54.842350 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:54.869480 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.870481 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.879673 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:54.880636 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:54.884752 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:54.885941 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:54.887382 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.892429 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.923930 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:54.924089 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:04:54.933768 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.938678 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.940050 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:54.956413 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:54.958148 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:54.981766 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.984532 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:04:55.014307 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.019142 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.020334 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.023988 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.027386 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.028395 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.077095 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:55.078768 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:55.095423 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.111855 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.128813 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.129839 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.136292 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.138078 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:55.151566 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:55.154204 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:55.195062 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:55.232522 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:55.232566 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:55.238197 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.243903 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.249154 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.253881 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:55.260458 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:55.264967 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.265265 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.275084 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.280643 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.281615 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.290563 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.305241 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.308501 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.318687 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.319062 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.342417 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.347262 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.351712 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.364244 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.364557 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:55.383347 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.392987 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:55.393749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:55.476974 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:55.495769 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.497053 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.508735 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.513669 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.518132 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.519215 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.523034 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.524900 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.533252 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.545224 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.545440 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.562951 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.564029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.564263 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.566679 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:55.588809 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:55.591244 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.592214 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.594863 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.611135 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.617525 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.643543 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:55.686178 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:55.753394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.757627 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.760886 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.763676 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.778395 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.794310 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.811329 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.813803 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.816357 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:55.822994 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:55.832401 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.835058 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.839004 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.842681 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:55.849308 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.856943 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.866335 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.883441 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.901072 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:55.915234 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.919744 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.920189 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:55.926991 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.927039 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:55.933071 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:55.933856 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:55.951452 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:55.988415 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:55.992074 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:56.022274 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:56.024560 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.107492 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:56.113173 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:56.113684 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.124361 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.126945 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.135432 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.174193 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.176530 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.177793 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.187495 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.188485 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.189826 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.193114 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.200840 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.212884 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:56.234447 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:56.278080 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.282249 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:56.283147 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:56.291735 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.297062 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.305207 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.319900 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.325026 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.350942 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.353207 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.358635 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.364684 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.368989 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.376961 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.377856 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.454051 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.455729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.466089 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.483179 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.485960 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.495549 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.498211 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:56.511611 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:56.513406 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:56.546617 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.552742 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.556509 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.559311 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.561830 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.562146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.607375 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.626147 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.643991 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.645269 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.650558 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.653184 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.654578 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.672379 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.679868 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:56.682910 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:56.693753 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.711481 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.712713 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.715186 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.718317 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.730273 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:56.774185 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:56.792963 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:56.814392 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.821485 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.823030 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.824098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.842053 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.849202 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:56.854408 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.855551 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:56.885005 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.886837 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.895709 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:56.895799 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:56.896016 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.897284 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:56.975489 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:56.976648 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.004539 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.008978 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.018848 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.019515 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.028350 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.030285 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.038126 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.059154 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.060166 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.062999 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.068810 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.094316 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:57.097441 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:57.118106 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:57.155299 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.156183 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.157384 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.167621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.169144 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.169150 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.183933 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:57.184329 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:57.214444 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.222479 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.223095 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.225458 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.229079 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.233891 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.257448 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.324203 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.332830 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.333345 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.361163 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.365964 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.368518 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.368643 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.369228 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.370872 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:57.379033 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:04:57.381798 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.389698 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.405694 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.412210 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.427975 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.504343 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.507066 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.507324 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.513221 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.513371 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.513866 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.544077 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:57.583904 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:57.586390 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:04:57.602358 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:57.613642 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.618093 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.624821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.632642 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.635855 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.637367 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.641650 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.642508 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.647647 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.657505 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:57.658447 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:04:57.659418 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.661725 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.688253 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:57.714185 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:04:57.717876 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:57.727152 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:04:57.746932 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.748747 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.755825 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.759117 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.771391 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.775312 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.794814 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:57.869976 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.871834 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:57.877027 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.878535 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.885749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.889230 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.891309 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.896970 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.905741 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.928602 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:57.938584 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.939246 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.939963 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:57.952270 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 08:04:57.965013 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:57.966459 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:57.967914 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.970081 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:57.976706 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:57.980957 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:57.982426 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:57.990326 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:57.995198 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.017143 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.018041 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:58.067662 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:58.074750 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.074748 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.079468 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.084378 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.101511 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.107578 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.124294 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.222084 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.226420 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.232268 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:58.234619 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:58.248525 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.248550 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.263007 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.263127 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.277513 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.284085 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.287294 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.288254 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.289837 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.299080 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.302839 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.304250 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.317934 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.333905 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.338603 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.361389 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.363665 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:04:58.364007 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:04:58.383744 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:04:58.404474 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.411130 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:58.414240 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:58.422160 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.425738 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.429658 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.438130 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.445474 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.451651 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.552676 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:58.554203 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:58.554975 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.562829 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.576921 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.577992 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:58.578527 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:58.593335 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.599474 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.604310 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.610860 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.618640 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.619458 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.623922 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.641089 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.649930 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.717154 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.720026 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.723450 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.727430 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.731189 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.744408 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.749529 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:58.760056 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:58.788563 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:58.816121 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.816260 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.819836 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.833824 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.840039 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.851195 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.861819 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.865409 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:58.875875 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.879313 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:58.884241 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:58.893280 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:58.905709 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:58.916909 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:58.993544 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.001505 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.012194 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.018602 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.021169 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.043093 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:59.046378 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:59.047248 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.116487 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.123342 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.125439 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.132107 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.136837 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.141474 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.145253 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.149942 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.155988 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.158833 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.160109 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.172353 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.181329 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.182504 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.183331 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.227030 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.227443 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.229362 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.243062 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.243276 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:04:59.254684 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:59.261678 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:59.281915 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.360042 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.363874 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.378119 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.383273 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.385816 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.386672 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.390470 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.392726 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.405344 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.412227 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.421386 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.447131 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.448257 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.449635 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.482376 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.488340 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:04:59.503077 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:59.504635 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.505528 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.507849 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.516759 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.536971 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:04:59.556530 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:59.601197 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 08:04:59.622519 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.631479 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.632983 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.633378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.633690 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.641749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.651751 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.653970 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.654032 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.657307 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.657896 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.658179 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.677103 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.731222 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.751924 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.753129 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.754463 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.763401 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.767427 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.767840 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.768834 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:04:59.807955 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:04:59.839112 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.846516 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.851343 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.863832 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:04:59.865047 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.873186 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.899833 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.902858 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:04:59.909356 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:04:59.916090 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:04:59.918268 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:04:59.920790 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:04:59.921101 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:04:59.923378 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:04:59.940317 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:00.016492 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:00.018064 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.033365 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.037650 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.039403 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:00.042711 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:00.068612 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.070267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.071341 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.077392 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.085733 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.088547 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:00.099337 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:00.099404 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.103410 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.111351 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.117601 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:00.140486 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:05:00.175807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.178058 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.178397 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.191282 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:00.193099 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:05:00.193727 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:00.200832 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:00.201637 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:00.204982 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.216912 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:00.255714 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:00.277876 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.280258 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.295702 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.296370 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.315599 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:00.322018 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:00.343024 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.343155 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.389883 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:00.393754 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:00.400735 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.401082 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.401448 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.403013 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.409446 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.436573 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.441188 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.444267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.452701 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.462046 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.469805 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.471374 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.490832 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.498909 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.499915 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:00.501490 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:00.502378 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.504923 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.506136 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.522269 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.535034 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:00.597221 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:00.656420 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.664502 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.670850 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.673725 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.676460 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.676966 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.709759 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.715747 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.720088 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.732766 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.734187 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.736389 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.743646 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.746676 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:00.754348 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:00.759694 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:00.761890 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.777597 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.779587 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.781634 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:00.792634 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:00.795123 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.811349 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.827798 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.837258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:00.850605 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:00.854480 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:00.855996 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:00.873461 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:00.885561 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:00.997164 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:01.005851 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.007761 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:01.008002 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:01.012130 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.014734 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.020654 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.021616 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.024401 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:01.050285 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:01.053061 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.054003 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.069607 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.069749 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.072543 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.084836 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.085649 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.085666 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.100688 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:01.107558 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:01.110512 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.117532 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.136056 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:01.146673 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:01.147747 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.173641 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.187516 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.190239 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.191934 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.201907 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.202905 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.217438 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.220536 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.303332 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.306632 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.312738 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.313263 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.321255 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.337275 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.342751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.395731 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:01.396431 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:01.401376 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.403812 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.405449 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.413203 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.414804 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.416319 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.543730 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.550123 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.550446 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.552891 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.559566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.579617 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.582640 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.585417 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.591827 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:01.594400 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:05:01.595073 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:01.595326 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.601089 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.602900 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.612262 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.622782 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.623001 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.625633 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.630236 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.631156 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:01.631314 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:01.638451 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.651762 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.691550 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.805394 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:01.808909 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:01.809181 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.813899 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.826299 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.829499 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.831066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.833335 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.834146 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.835631 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.848466 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.853827 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.874065 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.877310 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.903587 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.939599 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.943686 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:01.950743 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:01.964725 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:01.966159 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:01.966158 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:01.975172 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:01.991706 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.001103 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.002848 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.023643 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.024369 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:02.031518 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:05:02.037099 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.042613 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.121740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.125609 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.128183 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.129719 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.137478 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.160987 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:02.162335 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:02.165407 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.226603 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.241461 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.242322 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.253523 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.254112 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.256882 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.260783 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.275056 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.278409 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.289418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.289567 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.295334 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.298141 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.334167 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.395212 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.399077 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.400409 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:02.406689 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:02.411625 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:02.413218 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.418452 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.422710 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.442867 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.483541 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.492136 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.502192 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.511185 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.514366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.516088 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.554280 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.560897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.566056 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.566892 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.568904 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.571970 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.580704 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.583112 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.653995 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.655880 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.657375 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.664388 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.665212 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:02.673029 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:02.679680 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:02.686653 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.690059 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.695935 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.705101 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.706858 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.729084 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.729724 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:02.732257 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:02.734892 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.758752 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.807305 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.812516 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.813748 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.815371 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.829000 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.836050 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.836772 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:02.870019 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:02.879482 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:02.900403 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.909327 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.916121 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.927747 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:02.932278 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.937975 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.940194 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.949892 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:02.964159 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.966877 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:02.967032 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:02.968798 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:02.989672 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:02.992419 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:03.026679 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:03.044015 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:03.063621 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.076912 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.077064 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.079067 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.080044 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.102782 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:03.110453 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:03.118870 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.163127 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.163317 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.164012 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.175967 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.180459 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.196017 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.206358 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.217790 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.222388 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.236317 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.237314 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.267634 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.276757 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.277415 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:03.292572 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:05:03.293360 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:03.319929 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:03.335376 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:03.373477 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.374166 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.374979 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:03.375138 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:03.377918 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.387176 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.407567 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.409972 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.416464 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.418745 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.429159 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.431078 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.443342 13182 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.443346 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.472988 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.478444 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.481364 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.482846 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.484395 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:03.495968 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:03.499338 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:03.510250 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:03.516270 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.536116 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.574143 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.575836 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.582700 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.596339 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.616657 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.617364 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.629182 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.695605 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.697772 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.698778 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.699200 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.709486 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.710116 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:03.711483 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:03.727890 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:03.728955 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:03.752096 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:03.774418 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.781862 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.788084 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.788291 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.791903 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.792239 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.792302 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:03.796936 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.803095 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.804801 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.805735 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.809024 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.827356 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.842200 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.926762 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:03.928067 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.929476 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:03.930255 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:03.934302 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:03.935020 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.938807 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:03.951802 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:03.993901 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:03.999219 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.004470 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.005136 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.008860 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.018890 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.043102 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.129621 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.130962 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.140187 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.145599 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:04.152544 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:04.163523 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.164181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.170428 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.231117 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.232059 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.234097 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.237550 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.258190 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.260704 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.261982 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:04.291465 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:04.308893 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.312815 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.319855 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.321735 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.328024 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.351339 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.365227 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:04.377085 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:04.380054 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.391410 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.394728 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.403313 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.412707 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.428583 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.429957 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:04.456032 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 08:05:04.469653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.481418 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.482208 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.494081 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.507781 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.515997 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.519982 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.611907 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.614449 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.622977 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.631799 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.632442 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.640911 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.647426 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.647881 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.657196 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:04.664603 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:04.675550 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.682498 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.687073 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.696580 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.697214 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.702147 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.703718 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:04.706053 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:04.710219 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.710959 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.721915 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.724819 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.730906 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.736637 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.745427 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.769531 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.770886 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.776682 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.800379 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.807760 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:04.812764 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:04.902292 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.905171 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:04.907545 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.915079 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.928875 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:04.932068 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.964704 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:04.981570 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:04.986038 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:04.988022 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:04.989756 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.021711 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.025388 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.039319 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.049767 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.062368 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.063444 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.063604 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.064203 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.079530 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.086338 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:05.097916 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:05:05.104497 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:05.125849 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.182983 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:05.209009 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.210994 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.211086 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.211337 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:05.224838 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:05:05.228447 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.228888 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:05.229041 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:05.232327 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.276654 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.278144 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.283335 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.284722 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.303639 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.305500 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:05.312804 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:05.317231 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.325426 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.326117 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:05.331823 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:05:05.342505 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.344275 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.344911 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.373347 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.403617 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:05.410300 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:05.459362 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.473412 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.473624 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.476440 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.491312 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.492942 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.503623 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.504699 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.518036 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.518527 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.520871 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.523887 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.548372 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.549396 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:05.579406 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:05:05.606992 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:05.611028 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.613951 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.620949 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.621203 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.623148 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.636202 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.652920 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:05.673590 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.677038 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:05.702101 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:05.702399 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.714238 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.715656 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.720924 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.728767 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.740811 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.751859 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.756302 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.769444 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.776163 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.776530 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.779064 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.783741 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.784123 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:05.793619 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.803452 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.809373 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.812932 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:05.820139 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:05.826751 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.842746 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.847429 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:05.847558 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:05.964363 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:05.968240 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.968772 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:05.969128 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:05.973026 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:05.980969 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:05.986212 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.054267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.058255 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.071558 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.082906 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.095018 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.103720 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.107694 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.116457 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.123095 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.131067 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.133792 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.140692 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.159085 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.160040 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.176190 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.187132 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.202106 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.203172 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.203686 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:06.207242 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:06.213277 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.213408 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:06.261190 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.281211 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:06.302392 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:06.312171 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.313663 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.316826 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.320334 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.336369 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.353888 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.355779 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.396340 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.397871 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.397910 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:06.399856 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:06.400902 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.418907 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.426656 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.426782 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.449925 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.451210 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.455852 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.471030 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.477713 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.478485 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.479977 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.578101 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.579689 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.585284 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.585533 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.613974 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.617187 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.618351 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.619302 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.619719 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.634505 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.641165 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.649493 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.651196 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.672241 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:06.694783 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:06.772671 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.773120 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.776175 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.795928 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.802700 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.811553 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.813082 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.820961 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.825383 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.835538 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.835741 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:06.836001 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:06.845517 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.872292 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:06.877357 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 08:05:06.884747 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.932261 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.946667 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.949613 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:06.954978 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:06.973822 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.976492 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:06.981593 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:06.988533 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:06.989686 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:06.991091 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.017639 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.017939 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.022694 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.024046 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.025939 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.031431 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.032356 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.035312 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.052063 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.090210 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.092751 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:07.098805 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:05:07.104745 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:07.123622 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:07.162443 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.166326 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.166361 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.167407 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.173341 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.187628 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.189906 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.288954 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.303316 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.307629 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.309247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.313978 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.317994 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.319623 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.319947 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.324410 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.328702 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.333119 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.343431 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.389554 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.389971 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.396087 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.398020 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.399930 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.402015 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:07.408085 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:07.421891 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.430399 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:07.449581 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:07.457155 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.533612 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.539386 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.541334 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:07.542130 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:07.552765 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.560845 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.566664 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.578542 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.584816 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.585952 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.593667 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.598224 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.602066 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.611800 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.621819 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:07.652004 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:07.652478 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.654949 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.662731 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.669921 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.678239 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.678258 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.680912 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:07.690869 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:07.812860 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.813410 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.814535 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.814742 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.832304 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.839097 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.847079 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.868163 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:07.872260 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.876756 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.879791 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.888595 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.901613 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.902138 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.907754 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.926010 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.932637 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:07.937683 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:07.938129 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.943500 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:07.948459 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:07.954527 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:07.964125 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:07.971468 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:07.990154 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:07.993219 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:07.995853 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.007031 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:08.019032 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:08.019688 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.027248 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:08.030409 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:08.042377 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:08.149432 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.153391 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.160046 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.184679 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:08.186511 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:08.188640 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.194118 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.203703 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.206358 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.209014 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.219991 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.224979 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.233932 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:08.242239 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:08.247363 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.252080 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.262133 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.266480 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.280853 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.283592 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.291728 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.297499 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.299503 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.303519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.309368 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.324030 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.331305 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.333559 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.341431 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:08.352489 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:08.352798 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:08.440402 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 08:05:08.441213 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:08.453459 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.457024 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.457828 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.469296 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.470165 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.472640 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.480432 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.515926 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.519348 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.520044 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.538422 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.539355 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.540247 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.540247 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:08.542465 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:08.563170 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:08.664566 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.664659 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.671517 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.674405 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.674557 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.707859 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.711309 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.741353 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.755165 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:08.759532 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:05:08.759822 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.761248 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:08.771824 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:08.772053 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.809298 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:08.818867 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.825744 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.826692 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.827327 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.830382 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.840516 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:08.853822 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:08.856300 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:08.859763 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:08.982769 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:08.986739 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:08.996515 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:08.998349 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.005328 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.012378 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:09.013058 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:09.014879 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.049366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.052372 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:09.054107 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:09.055536 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.057864 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.060992 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.061254 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.062417 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.087949 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.093609 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.099481 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.117697 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.125556 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.134975 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.145782 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.159469 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.164022 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.168509 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.188220 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.199893 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.207304 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.218344 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:09.251834 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 08:05:09.304196 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.304400 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.305887 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.305925 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.306267 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.313553 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.315196 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.392019 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.392577 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.409389 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.421229 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.422142 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.422396 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.424369 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:09.450817 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:09.451870 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.454067 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.459976 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.462509 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.463019 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.479954 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:09.482537 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:09.483192 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.585742 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.588691 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.606169 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.612699 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.613445 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.614024 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.635972 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:09.647558 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:09.663547 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.668596 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.669878 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.672973 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.676569 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.692512 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.706281 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.722860 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.723656 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.748699 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.749694 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.749801 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.768062 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.769637 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.786705 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.798594 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.801084 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.804678 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.809926 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.821188 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.849030 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:09.906857 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.907577 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:09.908703 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:09.920337 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:09.923074 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:09.924990 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:09.945278 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:09.971833 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:09.988235 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.004967 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:10.005200 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:10.009693 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.012502 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.014724 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.016947 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.027980 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.050439 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.053436 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:10.055749 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:10.071678 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.078732 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.080745 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.085287 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.099622 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.159857 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.160733 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.167280 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:10.193296 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:10.194898 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.196333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.196545 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.197821 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.199493 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.203755 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.217110 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.219985 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:10.257241 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:10.258986 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.267875 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.268049 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:10.284247 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:10.316965 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.325435 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.330417 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.331040 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:10.333155 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:05:10.339171 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.348706 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:10.360998 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:10.388913 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:10.403856 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:10.421737 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.426167 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.428927 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.455951 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.478055 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.485504 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.486742 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.488256 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.498279 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.504722 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.507885 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.521464 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.530858 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.538383 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:10.540375 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:05:10.548110 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:10.565253 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:10.594733 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.602070 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.606379 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.606904 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.607628 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.613483 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.644210 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.646939 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.655520 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.656180 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.663539 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.666688 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:10.672369 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:10.680234 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.707907 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.777922 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:10.778996 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:10.779429 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.782800 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.788115 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:10.789188 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:10.789404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.792204 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.810581 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.817301 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.819316 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.822026 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.823801 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.824309 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:10.829699 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:10.835822 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.863487 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:10.873380 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:05:10.892632 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:10.926532 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:10.927867 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:10.930569 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:10.933980 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:10.937455 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:10.945989 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:10.953613 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.052070 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.054667 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.063445 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.077091 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.078353 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.092381 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.101855 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.110334 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.111640 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.113986 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.120333 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.137102 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.138173 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.147518 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:11.153070 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:11.163013 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.170370 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.181325 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.183974 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:11.186807 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:11.187917 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.195806 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.207504 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:11.249032 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:11.275147 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.277560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.282156 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.283474 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:11.310665 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:11.329180 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.329355 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.332026 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.360002 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.360235 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.362720 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.363360 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.372828 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.389838 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.396459 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.455330 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.458372 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.471050 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.474860 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.487847 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.489998 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.499612 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.578557 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.583323 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.591938 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.592767 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.594254 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.600730 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.605990 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.617063 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.618263 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.625952 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:11.643404 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:11.655295 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.655532 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.661741 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.676834 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:11.694473 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:11.706251 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.707891 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.714138 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.716780 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.722209 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.723457 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.758729 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.803977 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.805177 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.808758 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.813316 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.825388 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.829653 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.852416 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:11.895285 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:11.900743 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:11.903290 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:11.904258 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:11.913897 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:11.914171 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:11.919143 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:11.949086 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:11.977522 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:11.997378 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.003353 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.018334 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.030889 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.031399 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.050100 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.055876 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.128830 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.143530 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.143906 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.158713 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.163359 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.175257 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.177593 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.178290 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.180379 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.185649 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.200037 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.210471 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.227808 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.235499 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.238306 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.245496 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.254402 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:12.262629 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:12.273393 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.283329 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.299739 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.301911 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:12.358364 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:12.378544 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.380601 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.383708 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.389572 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.398144 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.407032 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.408169 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.441327 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.442994 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.446147 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:12.454625 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:05:12.455109 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.473378 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.474385 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.495847 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.513052 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.522511 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.524003 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.534389 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.538076 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.554662 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.575625 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.589020 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.596352 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.598250 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.605645 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:12.608135 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:12.627247 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.627818 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:12.629191 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:12.630359 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:12.657482 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:12.683531 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.685037 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.685395 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.693398 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.697566 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.707110 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:12.711483 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:12.738458 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:12.753504 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:12.773875 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:12.787567 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.790381 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.793280 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.795624 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.806864 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.813279 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.820748 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.820803 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.821238 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:12.833781 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.842389 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.844648 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.849740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:12.870630 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:12.873188 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.956750 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:12.957262 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:12.970405 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:12.970975 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.984484 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:12.984684 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:12.986660 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.079396 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.079838 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.097887 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.104117 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.107340 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.111361 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.118986 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.132544 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.135612 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.142871 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.154103 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.154138 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.154749 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.155427 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:13.179298 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:13.184971 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:13.188691 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.190977 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.194420 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.198824 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.199398 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:13.204658 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:13.226214 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.232769 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.268266 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.269130 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.269678 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:13.292281 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:13.293509 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.295913 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.314110 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.324491 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:13.396041 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:13.400288 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:13.423710 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.424420 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.428256 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.434581 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.436728 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.453207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.453719 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.454644 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.456854 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:13.474961 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:13.476078 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:13.488646 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:05:13.489238 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.512426 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.515676 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.544000 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.550498 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.550768 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.555639 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.556767 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.560497 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.581701 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.585966 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.594347 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.599095 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:13.603679 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:13.614095 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.636099 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.653116 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.655556 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:13.659339 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:13.682533 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.732950 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.734246 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.740278 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.741530 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:13.750788 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:13.756683 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.761066 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.764675 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.767119 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.776171 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:13.778052 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:13.780889 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.791558 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.792994 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.802435 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.823668 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:13.859122 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:13.898643 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:13.900136 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.902523 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.903038 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:13.908458 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:13.915769 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.931783 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.987027 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:13.989567 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:13.990381 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:13.997007 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.003851 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.012176 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.026968 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:14.035666 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 08:05:14.081826 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.101703 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.110644 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.113282 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.116909 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.118978 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.127859 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.132828 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.134397 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.143759 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.144497 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.163044 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.169070 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.193074 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:14.213141 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:14.289126 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.290791 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.296183 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.303658 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.307608 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.318303 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.321172 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:14.353098 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:14.363729 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.367540 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.372599 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.377777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.393987 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.395457 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.433440 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.439627 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.444203 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.445629 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.460682 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.467947 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.475083 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.476682 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.479801 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.484500 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.487479 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.487499 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.492676 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.497764 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.502523 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:14.572465 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:14.603170 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.604324 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.607208 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.614565 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.620196 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.631429 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.641599 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.690866 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.692523 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.702214 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.710155 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.716553 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.717126 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.732370 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.732743 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.736099 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.737522 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.753845 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.775305 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.781734 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.792886 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:14.799635 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:14.804131 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:14.914116 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.921029 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.923264 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.926771 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.927191 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.935007 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.939672 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.944002 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.946144 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:14.952160 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:14.954015 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:14.973388 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:14.991927 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:14.994271 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:14.998256 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.003098 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.011067 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.013037 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.031258 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.033162 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.081168 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:15.101465 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:15.177809 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.179523 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.191141 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.192800 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:15.202456 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.207080 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:15.207828 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.211985 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.212416 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:15.216627 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:15.219935 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.220299 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.223259 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.231544 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.256834 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.261741 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.268971 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.353747 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.355077 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.365731 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.371102 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.372224 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.376217 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.384567 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.395077 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:15.399971 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:15.400592 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.400959 14507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.401168 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.401868 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:15.403064 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:15.405248 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.430079 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:15.444492 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 08:05:15.451189 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.454110 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.457453 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.468894 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.474340 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.486009 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:15.492321 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:15.495332 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:15.496009 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:15.615816 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.618809 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.621807 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.627568 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.631721 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:15.637244 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:05:15.640892 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:15.641306 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.641616 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.642366 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.642586 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.644907 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.656021 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.673563 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.675192 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.719259 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.737085 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.743400 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.752060 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.757351 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:15.760289 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:15.763552 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.769064 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.781543 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.870154 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.875453 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:15.884860 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:15.895704 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.901155 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.905102 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:15.911831 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:15.912410 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:15.923609 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:15.941498 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:15.953234 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:15.962461 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:15.967859 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:15.970081 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:15.985532 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:15.990159 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:15.991465 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.003336 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:16.004592 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:16.006888 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 08:05:16.021756 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.033660 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.038592 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.038908 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.059022 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.062636 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.063654 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:16.066983 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:16.067906 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:16.084975 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:16.101835 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.109580 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.117892 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.117916 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.118321 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.122742 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.215207 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.223650 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.242465 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.246245 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.253988 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.254891 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.264222 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.268054 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.278259 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.278537 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.279268 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.287070 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.292199 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:16.296674 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 08:05:16.326685 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.423750 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.425194 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.425364 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.429440 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.431118 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.438709 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.439066 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.440565 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.463050 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.469777 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.474148 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.477854 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.483400 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.500591 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.543152 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.545825 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.549670 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.550983 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.551959 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.553649 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.556591 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:16.570139 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:16.588703 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:16.658089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.661273 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.668965 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.669140 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.673745 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.698333 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.702512 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.736351 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.749260 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.755825 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.759569 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.767938 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.779381 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.796111 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.796290 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.816073 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.820127 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.824434 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.831605 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.838367 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.876543 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.915112 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:16.921631 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:16.928683 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:16.929255 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:16.932507 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:16.944293 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:16.958508 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:16.975355 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:17.017850 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.025753 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.026046 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.026973 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:17.047081 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:17.055002 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.060756 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.061975 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.067673 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.090584 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.091131 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.092707 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.106547 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.119194 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:17.137231 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:17.153496 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.218897 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.219753 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.229292 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.230072 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.234205 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.257293 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.280784 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.291987 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.297420 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.302021 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.307919 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.310791 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.321189 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.322608 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.361373 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.377610 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.382376 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.386690 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.391820 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.404785 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.412009 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.413059 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.416118 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.428190 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.429589 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.433089 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.445027 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:17.460790 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:17.467087 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:17.556190 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:17.556841 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.557873 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.558454 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.558770 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.561817 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.566988 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.618009 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.638738 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.645702 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.646018 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.646901 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.647420 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.655454 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.660091 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:17.663934 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:17.679476 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:17.695226 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:17.704341 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:17.706043 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.717449 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:17.720679 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:17.724689 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.735620 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.736459 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.743844 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:17.747499 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:17.790966 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.800045 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:17.808538 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.809998 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.810632 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.812098 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.827953 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:17.919116 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.919116 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.924185 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:17.934295 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.938097 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:17.939540 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:17.956346 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:17.962385 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.976393 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:17.990319 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:17.994156 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.002979 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.008250 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.015478 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.016507 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.019079 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:18.037204 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:18.046491 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:18.051143 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:18.056824 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.091843 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.102782 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.107880 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.112010 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.112748 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.120950 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.123037 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.133240 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.146672 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.149014 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.150422 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.174202 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.238348 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.248703 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.248944 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.250660 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.253206 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.255344 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.263862 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:18.327327 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:18.331014 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.334964 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.345894 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.346153 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.356612 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.374331 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.382792 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.385890 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.388914 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.390655 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.419107 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:18.421065 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:18.424172 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.426717 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.432330 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:18.468729 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:18.501250 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:18.506345 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.517760 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.517757 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.519572 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.520037 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:18.528801 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:18.533515 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.535792 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:18.551379 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 08:05:18.551957 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:18.579715 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:18.620141 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.625519 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.626246 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.631661 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.643455 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.644155 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.658176 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:18.690543 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:18.700433 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.700702 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.705595 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.707741 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.708812 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.714241 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.715242 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.723129 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:18.723592 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:18.723784 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.743899 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.750048 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.763556 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.766749 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.771447 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:18.820526 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:18.853778 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:18.854506 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.857219 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.868057 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.871434 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.874819 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:18.884469 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:18.952742 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:18.973560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:18.975175 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.981173 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:18.987421 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:18.991250 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:18.995579 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.003820 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.033130 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.036670 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.044703 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.049031 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:19.053941 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:19.097576 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.097903 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.116828 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.123714 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.125716 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.131665 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.135967 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.138707 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.159606 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.162371 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.221680 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.227349 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.229410 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.229823 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.238423 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.249009 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.259001 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.313472 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.318495 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.319294 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.323858 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.328194 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:19.335386 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:19.349656 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.372809 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.423311 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.432519 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.433880 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.434018 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.443266 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.452108 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:19.471458 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:19.479121 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.536108 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.537950 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.544230 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.548734 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.551017 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.562496 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.569875 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.573901 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.577908 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.593906 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.596587 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.609804 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.611284 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.621385 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.636772 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.646307 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.658589 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:19.665555 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:19.667057 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.668442 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.680181 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.695376 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.813820 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:19.816994 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:19.859119 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.866973 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.869001 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.871346 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.876444 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.892222 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.893358 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.897323 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.907474 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.922580 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.927886 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:19.936401 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.941320 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:19.948350 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:19.949445 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:19.954634 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:19.955760 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:19.959738 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:19.975126 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.987116 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:19.996526 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.000645 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.001403 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.003394 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.003958 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.007419 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.037577 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.039624 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.040359 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.040887 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.051743 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.055330 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.066888 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.077634 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.091571 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:20.126652 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.200420 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.211989 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:20.230104 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.230368 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.245834 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:20.246912 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:20.247459 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.247768 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.248068 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.273265 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.286260 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.286801 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.289317 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.290802 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.303074 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.304029 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.305398 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:20.336639 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:20.336724 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:20.397491 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:20.463620 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.466404 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.474541 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.475193 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.476944 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:20.489185 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:20.499840 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.509516 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.510015 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.516671 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.517459 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.523412 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.532801 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.537045 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.543598 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.563109 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.563469 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.565680 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:20.576799 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 08:05:20.584543 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.586961 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.589469 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.615386 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:20.653366 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:20.654889 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.657366 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.672725 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.675742 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:20.676178 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:05:20.683012 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.686708 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.697412 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:20.708513 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 08:05:20.713512 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:20.713510 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:20.719067 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.733137 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.734244 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.735317 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:20.739563 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:20.740851 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.743115 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:20.805872 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:20.814566 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:20.838234 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.840902 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.846225 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.864531 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.871678 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.887182 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.909224 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.927913 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.931926 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.937214 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.942008 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.947438 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:20.950817 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.958566 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:20.961548 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.962438 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.974503 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:20.976097 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:20.976830 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:20.983063 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:20.999074 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:21.034416 14591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:05:21.056947 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:21.064560 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 08:05:21.121114 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.121397 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.123904 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.125542 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.143080 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:21.148998 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:21.152537 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.169184 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.173249 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.188073 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.189807 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:21.190522 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:21.199923 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.230299 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.237695 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.239699 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.258123 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.269150 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.281035 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.281621 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.289228 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.293169 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.304409 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:21.312705 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.359951 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:21.363036 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.369122 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.373852 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.374819 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.383041 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.405486 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.406888 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.456092 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.465440 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.466981 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.467523 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.484318 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.484819 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.502008 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.562781 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:21.578794 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:21.581300 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.586175 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.587786 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.588688 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.596271 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.618733 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.642372 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.644327 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.648963 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.649654 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.659080 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.663085 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.664209 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.726882 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.736142 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.737850 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.743788 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.745172 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.757552 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.765554 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.812809 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:21.816287 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.832218 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.833981 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.841569 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.844735 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.856563 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.899293 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.903646 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.908893 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:21.911082 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.917090 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.919296 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:21.927484 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:21.931847 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.983256 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:21.987763 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:21.989250 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:21.992508 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:21.999225 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.001739 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.046290 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:22.074777 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:22.112586 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.119752 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.136404 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.139151 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.156615 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.173843 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.175559 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.178673 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.187922 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.190975 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.193560 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.195559 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.203346 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.230202 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.287467 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.292876 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.305653 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.307857 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.309032 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.311394 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.329514 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.337015 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.338434 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.339836 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:22.346302 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:22.355434 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.361673 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:22.370088 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:22.387542 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.396106 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.416595 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.430301 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.433279 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.435974 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:22.456110 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:22.458740 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.461889 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.477805 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:22.501094 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:22.533108 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.543251 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.546624 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.546790 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.548049 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.573264 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.579767 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:22.579988 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.589496 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:22.636950 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.646519 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.659557 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.673734 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.678750 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.688345 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.688980 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.692163 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.699086 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.701646 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.708129 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.718387 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.728010 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.735373 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.744140 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.747922 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.762156 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.762430 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.768916 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:22.777344 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:22.792898 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:22.793499 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:22.802345 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:22.837091 13010 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:22.864544 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:22.876098 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.882088 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:22.884624 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.894512 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.897233 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.899055 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.916510 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:22.916574 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.940011 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:22.940277 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:22.943301 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:22.951509 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.952759 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:22.954116 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:22.992700 12879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:23.021342 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.023546 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.024740 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.033392 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.041592 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.058001 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.071115 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:23.087110 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:23.096244 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:23.149631 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.150987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.154470 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.161293 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.185444 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.186314 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.191190 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:23.194653 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:23.205796 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.208293 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.215651 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.221252 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.236439 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.242497 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.249519 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:23.262151 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:23.285404 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.286121 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.294317 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.320179 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.321269 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.322531 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.322788 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:23.340237 14460 consensus_peers.cc:485] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:23.390100 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.395056 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.398002 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.399631 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.399916 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.417984 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.427731 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.505928 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.507306 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.509399 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.514982 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.519359 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.528008 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:23.543130 13010 consensus_peers.cc:485] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:23.543183 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.553051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:23.570186 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 08:05:23.571478 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:23.574460 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.574910 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.591068 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.596279 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.597080 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.603619 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.609900 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:23.610065 14722 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:23.613828 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.617024 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.618250 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:23.627686 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 08:05:23.631999 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.647348 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.647665 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.711897 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.712973 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.713620 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.714605 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.723856 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.736363 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.753453 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.808102 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.808991 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.809736 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:23.813002 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:23.824359 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.824625 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:23.827693 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.829257 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:05:23.854071 13010 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:23.885377 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:23.885625 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:23.886235 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:23.886472 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:23.886973 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.914618 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:23.914824 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.027631 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.036418 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.037720 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.039590 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.049726 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.051167 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.052605 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.052877 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:24.055449 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:24.081687 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.095362 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.096150 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.109476 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.109715 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.113054 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.118667 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.120266 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.120829 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:24.130008 12748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:05:24.132007 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.132468 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.149484 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.155063 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.278873 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.290043 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.290043 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.293359 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.309103 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.309170 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.339463 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.360478 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.374095 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.378296 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.386848 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.389300 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.391070 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:24.410717 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:24.440896 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.453534 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.454984 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.462227 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.463111 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.465006 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.476686 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.484050 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.486178 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.496734 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.497202 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.508587 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.525151 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.531072 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.552747 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:24.590250 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:24.614486 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.626350 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.627774 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.632143 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.643769 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.645644 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.645808 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.693758 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.697139 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.704419 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.704716 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.722944 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.727470 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.749094 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.772598 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.782582 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.787849 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.788777 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.801774 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:24.829536 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:24.831224 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.832093 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.864025 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.864827 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.877173 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.881357 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.881989 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.896109 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:24.911265 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.931445 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:24.949291 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:24.980276 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:24.982760 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:24.982954 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.984216 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:24.987890 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.009897 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.031880 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.034011 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:05:25.035912 13141 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:25.037312 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.057974 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.064539 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:25.095620 14460 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:25.101368 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:25.108597 12879 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:25.108678 14591 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:25.184819 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.191952 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.194162 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.199424 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.200367 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:05:25.207921 13141 consensus_peers.cc:462] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:25.227108 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.236316 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.239894 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:25.251966 13010 consensus_peers.cc:485] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 08:05:25.256641 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.276582 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.276762 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.283414 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.288695 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.303031 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.313284 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.324987 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 08:05:25.325852 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:25.331861 13010 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:25.333040 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.333648 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.352628 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.364022 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.372295 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.377607 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.392299 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.400051 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.400784 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.420955 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.427335 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.431058 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.473287 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.490538 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.496017 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.496050 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.496731 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.496735 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.503155 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.534296 16680 raft_consensus.cc:998] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.128s)
I20240503 08:05:25.534828 16680 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:25.536350 14527 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:25.536739 13612 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.538662 16680 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.540561 14527 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.541383 12666 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
W20240503 08:05:25.557955 12657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
W20240503 08:05:25.561384 12657 catalog_manager.cc:4573] TS eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c917c5849f54411eb55d69ab33456bf4: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:25.563833 12946 consensus_queue.cc:237] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
W20240503 08:05:25.564733 12748 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:25.566116 14527 raft_consensus.cc:1261] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Refusing update from remote peer 0cf63591cfc244028e1fc368c2bca079: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:25.567202 16680 consensus_queue.cc:1035] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:25.569253 12881 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): Couldn't send request to peer 6c358da9faef48c4adb12cce628aeb38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c917c5849f54411eb55d69ab33456bf4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.570168 16680 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.572492 12660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c917c5849f54411eb55d69ab33456bf4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:25.573083 12666 catalog_manager.cc:5484] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "0cf63591cfc244028e1fc368c2bca079" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.575239 14527 raft_consensus.cc:2932] T c917c5849f54411eb55d69ab33456bf4 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.595747 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.600979 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.602169 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.609555 15586 raft_consensus.cc:998] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.179s)
I20240503 08:05:25.609946 15586 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:25.612293 14656 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:05:25.612934 13076 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:05:25.614245 15586 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:25.614504 15586 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:25.616048 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.616178 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.616515 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.634649 16337 raft_consensus.cc:998] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.013s)
I20240503 08:05:25.635568 16337 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:25.645898 12815 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:25.646296 16337 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.647692 13206 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:25.648169 16337 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.650517 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.662993 16337 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.663300 12815 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.665376 13206 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.665791 12665 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.671310 15586 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.673417 13076 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.674281 12665 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.676631 14656 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.678462 6559 raft_consensus.cc:998] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.096s)
I20240503 08:05:25.678745 6559 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:25.681636 12946 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:25.682468 16733 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.684484 16733 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.688001 12665 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.690981 12946 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.698115 15281 raft_consensus.cc:998] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.104s)
W20240503 08:05:25.698333 13010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 08:05:25.698457 15281 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:25.699718 14656 consensus_queue.cc:237] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:25.699987 14658 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:25.700979 16773 raft_consensus.cc:998] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.039s)
I20240503 08:05:25.701328 16773 consensus_queue.cc:237] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:25.703089 14789 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.703513 16743 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.703492 15281 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:25.704973 12946 raft_consensus.cc:1261] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 8398e6f89bf5419d81ed5bed991da0cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:25.705107 14658 raft_consensus.cc:1261] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:05:25.705363 14593 consensus_peers.cc:485] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf -> Peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): Couldn't send request to peer 6c358da9faef48c4adb12cce628aeb38. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.705490 6559 consensus_queue.cc:1035] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.705601 16743 consensus_queue.cc:1035] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.707780 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.710950 6559 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.712364 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.712836 12658 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d8994c38d0f640e9bd5b5c322c603b9f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:25.715238 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.716082 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.718302 12665 catalog_manager.cc:5484] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "8398e6f89bf5419d81ed5bed991da0cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.722534 12946 raft_consensus.cc:2932] T d8994c38d0f640e9bd5b5c322c603b9f P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.726128 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.729812 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.750046 15586 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.751331 14658 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.752019 14789 raft_consensus.cc:2932] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.756611 15281 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.754766 12665 catalog_manager.cc:5484] T 1c84f7a57e0945d9a25a70798f054bd1 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.758142 14658 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.761765 12667 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 7 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.763175 15281 raft_consensus.cc:998] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.129s)
I20240503 08:05:25.763478 15281 consensus_queue.cc:237] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:05:25.764573 14789 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.765302 12946 raft_consensus.cc:1261] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.765566 15281 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.767634 16803 consensus_queue.cc:1035] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.768090 16803 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:25.770208 12672 catalog_manager.cc:5484] T 7d61aa445a50417b82916019c2a4298c P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:25.770316 14789 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:25.774423 13076 consensus_queue.cc:237] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:25.775030 16803 raft_consensus.cc:998] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.260s)
I20240503 08:05:25.775521 16803 consensus_queue.cc:237] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
W20240503 08:05:25.776036 13012 consensus_peers.cc:485] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 -> Peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847): Couldn't send request to peer 6c358da9faef48c4adb12cce628aeb38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 076781af2ac14ddc87d31e3ec77db08f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.776674 13206 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.777143 15281 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.777523 14656 raft_consensus.cc:1261] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.777534 14657 raft_consensus.cc:1261] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.778178 16803 consensus_queue.cc:1035] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.778471 16803 consensus_queue.cc:1035] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.781123 14657 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.783959 14657 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.783967 16842 raft_consensus.cc:2932] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.785694 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 076781af2ac14ddc87d31e3ec77db08f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:25.785880 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.785688 12667 catalog_manager.cc:5484] T 076781af2ac14ddc87d31e3ec77db08f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 8 to 9, VOTER 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:25.786075 16842 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.791172 13206 raft_consensus.cc:2932] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.794301 12673 catalog_manager.cc:5484] T 672fc9981d074f7e8f2324ba0b06be57 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:25.811829 12946 raft_consensus.cc:2932] T 7d61aa445a50417b82916019c2a4298c P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:25.832558 16510 raft_consensus.cc:998] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.022s)
I20240503 08:05:25.832949 16510 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:25.835287 13206 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:05:25.835660 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 08:05:25.836091 14460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:25.837153 16510 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:25.837472 16510 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:25.843390 16510 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.844910 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.845942 13206 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.846160 12667 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 6 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.847497 13076 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.848052 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.848718 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.859948 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.861095 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:25.866883 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.890523 16833 raft_consensus.cc:998] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.310s)
I20240503 08:05:25.891052 16833 consensus_queue.cc:237] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:25.891623 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.892639 14657 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.892706 14527 raft_consensus.cc:1261] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.893118 16803 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.893323 16859 consensus_queue.cc:1035] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:25.900813 14722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 08:05:25.910219 16803 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.913311 14527 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 82cd33f2bae4419b946095f8eb93e1e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.913796 12666 catalog_manager.cc:5484] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.915647 14657 raft_consensus.cc:2932] T a60b868dcc504bb29d08e2752c8dd44d P 8398e6f89bf5419d81ed5bed991da0cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:25.951104 16833 raft_consensus.cc:998] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.550s)
I20240503 08:05:25.951517 16833 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:05:25.953265 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.953963 14789 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.954605 16858 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.956234 16858 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:25.965066 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:25.968195 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:25.970898 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.971467 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.982007 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:25.990087 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:25.991705 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:25.991974 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.005939 15281 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.008605 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.008914 12673 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.016474 16803 raft_consensus.cc:998] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.250s)
I20240503 08:05:26.016799 16803 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:26.018146 14657 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:26.018353 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.018548 16859 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.020850 14789 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.022333 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.025051 15281 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.026850 14657 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.030248 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.030989 16972 ts_tablet_manager.cc:902] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Initiating tablet copy from peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:05:26.032083 12670 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf reported cstate change: config changed from index 8 to 9, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.035023 13076 consensus_queue.cc:237] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:26.036391 14657 raft_consensus.cc:1261] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.036796 16859 consensus_queue.cc:1035] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:26.036955 13012 consensus_peers.cc:485] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e7f229898c24572815177628720521f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.044204 16972 tablet_copy_client.cc:308] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33937
I20240503 08:05:26.044468 16859 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.044790 12966 tablet_copy_service.cc:140] P 0cf63591cfc244028e1fc368c2bca079: Received BeginTabletCopySession request for tablet c917c5849f54411eb55d69ab33456bf4 from peer 6c358da9faef48c4adb12cce628aeb38 ({username='slave'} at 127.0.99.129:43169)
I20240503 08:05:26.044884 12966 tablet_copy_service.cc:161] P 0cf63591cfc244028e1fc368c2bca079: Beginning new tablet copy session on tablet c917c5849f54411eb55d69ab33456bf4 from peer 6c358da9faef48c4adb12cce628aeb38 at {username='slave'} at 127.0.99.129:43169: session id = 6c358da9faef48c4adb12cce628aeb38-c917c5849f54411eb55d69ab33456bf4
I20240503 08:05:26.046249 12966 tablet_copy_source_session.cc:205] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.046898 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9e7f229898c24572815177628720521f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.047317 12666 catalog_manager.cc:5484] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.047590 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c917c5849f54411eb55d69ab33456bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.049211 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.049850 14657 raft_consensus.cc:2932] T 9e7f229898c24572815177628720521f P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.073173 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.081844 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.099243 16972 tablet_copy_client.cc:783] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.106271 16972 tablet_copy_client.cc:647] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.107764 16972 tablet_copy_client.cc:523] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.110410 16972 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:05:26.113054 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.122655 15237 raft_consensus.cc:998] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.467s)
I20240503 08:05:26.123016 15237 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:26.124064 12946 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:05:26.124418 15237 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:26.130728 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.136240 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.137229 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.137576 16889 raft_consensus.cc:998] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.184s)
I20240503 08:05:26.137971 16889 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:26.139734 13076 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.140144 16887 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.140664 12946 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.141044 16887 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.146865 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.141865 16972 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.160475 16972 tablet_bootstrap.cc:492] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Bootstrap complete.
I20240503 08:05:26.160861 16972 ts_tablet_manager.cc:1367] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20240503 08:05:26.161916 16972 raft_consensus.cc:348] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.162822 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.163015 16972 raft_consensus.cc:729] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:05:26.163197 16972 consensus_queue.cc:260] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.164330 16972 ts_tablet_manager.cc:1398] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:26.164870 12966 tablet_copy_service.cc:342] P 0cf63591cfc244028e1fc368c2bca079: Request end of tablet copy session 6c358da9faef48c4adb12cce628aeb38-c917c5849f54411eb55d69ab33456bf4 received from {username='slave'} at 127.0.99.129:43169
I20240503 08:05:26.164953 12966 tablet_copy_service.cc:433] P 0cf63591cfc244028e1fc368c2bca079: ending tablet copy session 6c358da9faef48c4adb12cce628aeb38-c917c5849f54411eb55d69ab33456bf4 on tablet c917c5849f54411eb55d69ab33456bf4 with peer 6c358da9faef48c4adb12cce628aeb38
I20240503 08:05:26.168833 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.169911 16887 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.172622 13076 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.173216 12670 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.173431 12946 raft_consensus.cc:2932] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.174813 17021 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.177465 16972 ts_tablet_manager.cc:902] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:05:26.177891 16972 tablet_copy_client.cc:308] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:05:26.179627 12946 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.184248 12666 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:26.199730 12815 consensus_queue.cc:237] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:05:26.201117 12946 raft_consensus.cc:1261] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 08:05:26.201586 12750 consensus_peers.cc:485] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 -> Peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): Couldn't send request to peer b052e89b63c24258872c832a6a0d4d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 047aba81fad543ce8b562a184746b163. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.201704 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 076781af2ac14ddc87d31e3ec77db08f from peer 6c358da9faef48c4adb12cce628aeb38 ({username='slave'} at 127.0.99.129:47063)
I20240503 08:05:26.201812 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 076781af2ac14ddc87d31e3ec77db08f from peer 6c358da9faef48c4adb12cce628aeb38 at {username='slave'} at 127.0.99.129:47063: session id = 6c358da9faef48c4adb12cce628aeb38-076781af2ac14ddc87d31e3ec77db08f
I20240503 08:05:26.202756 17048 consensus_queue.cc:1035] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:26.203316 13097 tablet_copy_source_session.cc:205] T 076781af2ac14ddc87d31e3ec77db08f P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.204675 15237 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.204946 12946 raft_consensus.cc:2932] T 047aba81fad543ce8b562a184746b163 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.205873 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076781af2ac14ddc87d31e3ec77db08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.207075 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 047aba81fad543ce8b562a184746b163 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.207458 12670 catalog_manager.cc:5484] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } }
I20240503 08:05:26.208004 16972 tablet_copy_client.cc:783] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.212854 16972 tablet_copy_client.cc:647] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.214632 16972 tablet_copy_client.cc:523] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.216116 16972 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:05:26.226358 13206 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:26.227538 14657 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:26.228024 16745 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.229456 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4): 4838 bytes on disk
I20240503 08:05:26.230145 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(c917c5849f54411eb55d69ab33456bf4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.230823 16745 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.233359 12666 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 8 to 9, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.234012 14657 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.236737 16972 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.236987 16972 tablet_bootstrap.cc:492] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap complete.
I20240503 08:05:26.237147 16972 ts_tablet_manager.cc:1367] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20240503 08:05:26.237581 16972 raft_consensus.cc:348] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.237677 16972 raft_consensus.cc:729] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:05:26.237799 16972 consensus_queue.cc:260] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.237871 13056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba with delete_type TABLET_DATA_TOMBSTONED (TS 39827620daed48cb8f137442e1f1da76 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33562
I20240503 08:05:26.238051 16972 ts_tablet_manager.cc:1398] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:26.238777 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 6c358da9faef48c4adb12cce628aeb38-076781af2ac14ddc87d31e3ec77db08f received from {username='slave'} at 127.0.99.129:47063
I20240503 08:05:26.238863 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 6c358da9faef48c4adb12cce628aeb38-076781af2ac14ddc87d31e3ec77db08f on tablet 076781af2ac14ddc87d31e3ec77db08f with peer 6c358da9faef48c4adb12cce628aeb38
I20240503 08:05:26.248066 13206 consensus_queue.cc:237] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
W20240503 08:05:26.249879 13143 consensus_peers.cc:485] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e0d84a47cc34af9bdd6b955feb2c3ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.250766 14657 raft_consensus.cc:1261] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.251039 16972 ts_tablet_manager.cc:902] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Initiating tablet copy from peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483)
I20240503 08:05:26.251124 16743 consensus_queue.cc:1035] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.251425 16972 tablet_copy_client.cc:308] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39483
I20240503 08:05:26.251821 14678 tablet_copy_service.cc:140] P 8398e6f89bf5419d81ed5bed991da0cf: Received BeginTabletCopySession request for tablet d8994c38d0f640e9bd5b5c322c603b9f from peer 6c358da9faef48c4adb12cce628aeb38 ({username='slave'} at 127.0.99.129:42235)
I20240503 08:05:26.251907 14678 tablet_copy_service.cc:161] P 8398e6f89bf5419d81ed5bed991da0cf: Beginning new tablet copy session on tablet d8994c38d0f640e9bd5b5c322c603b9f from peer 6c358da9faef48c4adb12cce628aeb38 at {username='slave'} at 127.0.99.129:42235: session id = 6c358da9faef48c4adb12cce628aeb38-d8994c38d0f640e9bd5b5c322c603b9f
I20240503 08:05:26.252732 17086 tablet_replica.cc:330] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: stopping tablet replica
I20240503 08:05:26.252696 16743 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.252866 17086 raft_consensus.cc:2229] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:26.253278 14678 tablet_copy_source_session.cc:205] T d8994c38d0f640e9bd5b5c322c603b9f P 8398e6f89bf5419d81ed5bed991da0cf: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.253293 17086 raft_consensus.cc:2258] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:26.254586 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0e0d84a47cc34af9bdd6b955feb2c3ba with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.254837 17086 ts_tablet_manager.cc:1875] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:26.255378 16972 ts_tablet_manager.cc:1875] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:26.255394 12666 catalog_manager.cc:5484] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.254804 14657 raft_consensus.cc:2932] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.257334 16972 ts_tablet_manager.cc:1888] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 08:05:26.257529 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8994c38d0f640e9bd5b5c322c603b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.259763 17086 ts_tablet_manager.cc:1888] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 08:05:26.259997 17086 log.cc:1196] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 39827620daed48cb8f137442e1f1da76: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e0d84a47cc34af9bdd6b955feb2c3ba
I20240503 08:05:26.260643 12657 catalog_manager.cc:4830] TS 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): tablet 0e0d84a47cc34af9bdd6b955feb2c3ba (table rebalance_test_table_9 [id=3f8dc7f8c13f4d338c817f912734bd17]) successfully deleted
I20240503 08:05:26.263787 16972 tablet_copy_client.cc:783] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.271461 16972 tablet_copy_client.cc:647] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.276953 16972 tablet_copy_client.cc:523] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.278461 16972 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap starting.
I20240503 08:05:26.286370 16753 raft_consensus.cc:998] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.250s)
I20240503 08:05:26.286767 16753 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:26.288749 14657 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.289392 12946 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.290367 16753 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.290613 16753 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.293812 16753 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.296414 12946 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.296569 12669 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.296636 14657 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.303927 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.313217 16972 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.313515 16972 tablet_bootstrap.cc:492] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Bootstrap complete.
I20240503 08:05:26.313694 16972 ts_tablet_manager.cc:1367] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.006s
I20240503 08:05:26.314155 16972 raft_consensus.cc:348] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.314273 16972 raft_consensus.cc:729] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Initialized, Role: FOLLOWER
I20240503 08:05:26.314417 16972 consensus_queue.cc:260] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:26.314704 16972 ts_tablet_manager.cc:1398] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:26.315536 14678 tablet_copy_service.cc:342] P 8398e6f89bf5419d81ed5bed991da0cf: Request end of tablet copy session 6c358da9faef48c4adb12cce628aeb38-d8994c38d0f640e9bd5b5c322c603b9f received from {username='slave'} at 127.0.99.129:42235
I20240503 08:05:26.315616 14678 tablet_copy_service.cc:433] P 8398e6f89bf5419d81ed5bed991da0cf: ending tablet copy session 6c358da9faef48c4adb12cce628aeb38-d8994c38d0f640e9bd5b5c322c603b9f on tablet d8994c38d0f640e9bd5b5c322c603b9f with peer 6c358da9faef48c4adb12cce628aeb38
I20240503 08:05:26.338826 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.343590 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.346071 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.350723 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.362524 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.376194 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.386083 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.386365 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.387053 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.408354 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.408766 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.411003 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.434538 12815 raft_consensus.cc:1203] T c917c5849f54411eb55d69ab33456bf4 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240503 08:05:26.475368 13141 consensus_peers.cc:485] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 -> Peer eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): Couldn't send request to peer eae9f53215a34bc88abe7b3c1c223319. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:26.483713 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.484661 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f): 4838 bytes on disk
I20240503 08:05:26.484833 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.485081 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(d8994c38d0f640e9bd5b5c322c603b9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.487038 12862 maintenance_manager.cc:419] P 6c358da9faef48c4adb12cce628aeb38: Scheduling UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f): 4582 bytes on disk
I20240503 08:05:26.487447 12752 maintenance_manager.cc:643] P 6c358da9faef48c4adb12cce628aeb38: UndoDeltaBlockGCOp(076781af2ac14ddc87d31e3ec77db08f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.492744 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.493616 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.512704 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.515436 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.527868 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.529559 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.551640 17244 ts_tablet_manager.cc:902] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:05:26.552876 17244 tablet_copy_client.cc:308] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:05:26.553520 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet 9e7f229898c24572815177628720521f from peer 82cd33f2bae4419b946095f8eb93e1e4 ({username='slave'} at 127.0.99.133:36949)
I20240503 08:05:26.553625 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet 9e7f229898c24572815177628720521f from peer 82cd33f2bae4419b946095f8eb93e1e4 at {username='slave'} at 127.0.99.133:36949: session id = 82cd33f2bae4419b946095f8eb93e1e4-9e7f229898c24572815177628720521f
I20240503 08:05:26.554626 13097 tablet_copy_source_session.cc:205] T 9e7f229898c24572815177628720521f P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.557345 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7f229898c24572815177628720521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.559898 17244 tablet_copy_client.cc:783] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.595166 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.602303 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.608819 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.609607 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.611261 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.621793 17244 tablet_copy_client.cc:647] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.627303 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.632808 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.634673 12815 raft_consensus.cc:1203] T d8994c38d0f640e9bd5b5c322c603b9f P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 08:05:26.634732 17244 tablet_copy_client.cc:523] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:05:26.636157 12657 catalog_manager.cc:4573] TS eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48e4f2a993164fb6b1df674311698006: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:26.636581 17244 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap starting.
I20240503 08:05:26.647164 17244 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.648121 17244 tablet_bootstrap.cc:492] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Bootstrap complete.
I20240503 08:05:26.648370 17244 ts_tablet_manager.cc:1367] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:05:26.649439 17244 raft_consensus.cc:348] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:26.649679 17244 raft_consensus.cc:729] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd33f2bae4419b946095f8eb93e1e4, State: Initialized, Role: FOLLOWER
I20240503 08:05:26.649883 17244 consensus_queue.cc:260] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:26.651072 17244 ts_tablet_manager.cc:1398] T 9e7f229898c24572815177628720521f P 82cd33f2bae4419b946095f8eb93e1e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:26.652276 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-9e7f229898c24572815177628720521f received from {username='slave'} at 127.0.99.133:36949
I20240503 08:05:26.652352 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session 82cd33f2bae4419b946095f8eb93e1e4-9e7f229898c24572815177628720521f on tablet 9e7f229898c24572815177628720521f with peer 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:05:26.660224 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.662714 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.678237 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.679751 14789 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:26.681335 14657 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:05:26.682377 17130 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:26.684562 17133 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.687126 12669 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 10 to 11, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.688501 14657 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:26.691254 14527 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:05:26.692600 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4f2a993164fb6b1df674311698006 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53588
I20240503 08:05:26.692662 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.693158 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:26.693420 17345 tablet_replica.cc:330] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:05:26.693559 16889 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.693612 17345 raft_consensus.cc:2229] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:26.693862 17345 raft_consensus.cc:2258] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:26.695540 17346 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.695854 13076 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.698215 12667 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
I20240503 08:05:26.699194 14574 maintenance_manager.cc:419] P 82cd33f2bae4419b946095f8eb93e1e4: Scheduling UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f): 4581 bytes on disk
I20240503 08:05:26.699595 14464 maintenance_manager.cc:643] P 82cd33f2bae4419b946095f8eb93e1e4: UndoDeltaBlockGCOp(9e7f229898c24572815177628720521f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.700922 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.704985 14789 consensus_queue.cc:237] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:05:26.705901 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.706197 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cc823a5ce34916894cfde0a6614bd1 with delete_type TABLET_DATA_TOMBSTONED (TS af712db2f85249adbe718e8ba0549f84 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33872
I20240503 08:05:26.709391 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.711329 14658 raft_consensus.cc:1261] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Refusing update from remote peer b052e89b63c24258872c832a6a0d4d4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:05:26.712028 16753 consensus_queue.cc:1035] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:05:26.713390 14527 consensus_queue.cc:237] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:26.714454 16753 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.715090 13076 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.715433 16889 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:26.716864 16889 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.716989 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 48e4f2a993164fb6b1df674311698006 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.717257 14657 raft_consensus.cc:2932] T 48e4f2a993164fb6b1df674311698006 P 8398e6f89bf5419d81ed5bed991da0cf [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.718362 12669 catalog_manager.cc:5484] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d reported cstate change: config changed from index 11 to 12, VOTER 39827620daed48cb8f137442e1f1da76 (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "b052e89b63c24258872c832a6a0d4d4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.718605 13076 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.719306 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 20cc823a5ce34916894cfde0a6614bd1 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.720620 13206 raft_consensus.cc:1261] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Refusing update from remote peer 82cd33f2bae4419b946095f8eb93e1e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:26.721153 12667 catalog_manager.cc:5484] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 reported cstate change: config changed from index 9 to 10, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "82cd33f2bae4419b946095f8eb93e1e4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.722841 17346 consensus_queue.cc:1035] T 20cc823a5ce34916894cfde0a6614bd1 P 82cd33f2bae4419b946095f8eb93e1e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.724354 13206 raft_consensus.cc:858] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
W20240503 08:05:26.724421 13206 raft_consensus.cc:1565] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 NON_PARTICIPANT]: Could not prepare op 'term: 2 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:26.724696 13206 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 NON_PARTICIPANT]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } }
W20240503 08:05:26.724823 14722 consensus_peers.cc:485] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d -> Peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153): Couldn't send request to peer 39827620daed48cb8f137442e1f1da76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e4f2a993164fb6b1df674311698006. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.728097 13206 raft_consensus.cc:1467] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 08:05:26.728770 13206 raft_consensus.cc:2932] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.732309 17345 ts_tablet_manager.cc:1875] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:26.733280 12815 raft_consensus.cc:1203] T 076781af2ac14ddc87d31e3ec77db08f P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:05:26.735982 17345 ts_tablet_manager.cc:1888] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:05:26.736255 17345 log.cc:1196] T 48e4f2a993164fb6b1df674311698006 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/48e4f2a993164fb6b1df674311698006
I20240503 08:05:26.736979 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet 48e4f2a993164fb6b1df674311698006 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
W20240503 08:05:26.737653 12657 catalog_manager.cc:4804] TS af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): delete failed for tablet 20cc823a5ce34916894cfde0a6614bd1 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 8 but the committed config has opid_index of 10
I20240503 08:05:26.748497 17349 ts_tablet_manager.cc:902] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Initiating tablet copy from peer 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129:43847)
I20240503 08:05:26.749279 17349 tablet_copy_client.cc:308] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43847
I20240503 08:05:26.749706 12835 tablet_copy_service.cc:140] P 6c358da9faef48c4adb12cce628aeb38: Received BeginTabletCopySession request for tablet 047aba81fad543ce8b562a184746b163 from peer b052e89b63c24258872c832a6a0d4d4d ({username='slave'} at 127.0.99.135:45337)
I20240503 08:05:26.749809 12835 tablet_copy_service.cc:161] P 6c358da9faef48c4adb12cce628aeb38: Beginning new tablet copy session on tablet 047aba81fad543ce8b562a184746b163 from peer b052e89b63c24258872c832a6a0d4d4d at {username='slave'} at 127.0.99.135:45337: session id = b052e89b63c24258872c832a6a0d4d4d-047aba81fad543ce8b562a184746b163
I20240503 08:05:26.751019 12835 tablet_copy_source_session.cc:205] T 047aba81fad543ce8b562a184746b163 P 6c358da9faef48c4adb12cce628aeb38: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.752418 17349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047aba81fad543ce8b562a184746b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.754861 17349 tablet_copy_client.cc:783] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.760299 17349 tablet_copy_client.cc:647] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.765061 17349 tablet_copy_client.cc:523] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.766893 17349 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap starting.
I20240503 08:05:26.776093 17349 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.776378 17349 tablet_bootstrap.cc:492] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Bootstrap complete.
I20240503 08:05:26.776576 17349 ts_tablet_manager.cc:1367] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:05:26.776948 17349 raft_consensus.cc:348] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:05:26.777045 17349 raft_consensus.cc:729] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Initialized, Role: FOLLOWER
I20240503 08:05:26.777474 17349 consensus_queue.cc:260] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } }
I20240503 08:05:26.777858 17349 ts_tablet_manager.cc:1398] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:26.779179 12835 tablet_copy_service.cc:342] P 6c358da9faef48c4adb12cce628aeb38: Request end of tablet copy session b052e89b63c24258872c832a6a0d4d4d-047aba81fad543ce8b562a184746b163 received from {username='slave'} at 127.0.99.135:45337
I20240503 08:05:26.779260 12835 tablet_copy_service.cc:433] P 6c358da9faef48c4adb12cce628aeb38: ending tablet copy session b052e89b63c24258872c832a6a0d4d4d-047aba81fad543ce8b562a184746b163 on tablet 047aba81fad543ce8b562a184746b163 with peer b052e89b63c24258872c832a6a0d4d4d
I20240503 08:05:26.832609 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.835891 14836 maintenance_manager.cc:419] P b052e89b63c24258872c832a6a0d4d4d: Scheduling UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163): 4923 bytes on disk
I20240503 08:05:26.836331 14726 maintenance_manager.cc:643] P b052e89b63c24258872c832a6a0d4d4d: UndoDeltaBlockGCOp(047aba81fad543ce8b562a184746b163) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.837591 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.838661 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.839066 14506 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:26.840341 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.860314 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.862534 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.871140 17244 ts_tablet_manager.cc:902] T 0e0d84a47cc34af9bdd6b955feb2c3ba P 82cd33f2bae4419b946095f8eb93e1e4: Initiating tablet copy from peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
F20240503 08:05:26.871332 17244 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e0d84a47cc34af9bdd6b955feb2c3ba.queue_time_us, tablet-download-pool-0e0d84a47cc34af9bdd6b955feb2c3ba.queue_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.queue_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.queue_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_cpu_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_cpu_time_us) (tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_wall_time_us, tablet-download-pool-20cc823a5ce34916894cfde0a6614bd1.run_wall_time_us) (tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.queue_time_us, tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.queue_time_us) (tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.run_cpu_time_us, tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.run_cpu_time_us) (tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.run_wall_time_us, tablet-download-pool-389b40c2136d45ab86ccf49df8d1b71e.run_wall_time_us) (tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.queue_time_us, tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.queue_time_us) (tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.run_cpu_time_us, tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.run_cpu_time_us) (tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.run_wall_time_us, tablet-download-pool-5212dff7ded143a7923bb6176466b0fa.run_wall_time_us) (tablet-download-pool-5480e544dca3428ab4fcf190577d5331.queue_time_us, tablet-download-pool-5480e544dca3428ab4fcf190577d5331.queue_time_us) (tablet-download-pool-5480e544dca3428ab4fcf190577d5331.run_cpu_time_us, tablet-download-pool-5480e544dca3428ab4fcf190577d5331.run_cpu_time_us) (tablet-download-pool-5480e544dca3428ab4fcf190577d5331.run_wall_time_us, tablet-download-pool-5480e544dca3428ab4fcf190577d5331.run_wall_time_us) (tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.queue_time_us, tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.queue_time_us) (tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.run_cpu_time_us, tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.run_cpu_time_us) (tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.run_wall_time_us, tablet-download-pool-5cee7c94dd0941d99d9dd6ceb39fc190.run_wall_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.queue_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.queue_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_cpu_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_cpu_time_us) (tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_wall_time_us, tablet-download-pool-70c5de226c7a4b349eef86a79d172851.run_wall_time_us) (tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.queue_time_us, tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.queue_time_us) (tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.run_cpu_time_us, tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.run_cpu_time_us) (tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.run_wall_time_us, tablet-download-pool-7c630e5cf7c0419cbf5d13413d22d317.run_wall_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.queue_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.queue_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.run_cpu_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.run_cpu_time_us) (tablet-download-pool-9e7f229898c24572815177628720521f.run_wall_time_us, tablet-download-pool-9e7f229898c24572815177628720521f.run_wall_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.queue_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.queue_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_cpu_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_cpu_time_us) (tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_wall_time_us, tablet-download-pool-a60b868dcc504bb29d08e2752c8dd44d.run_wall_time_us) (tablet-download-pool-a924487225dc4246aeb695060084096b.queue_time_us, tablet-download-pool-a924487225dc4246aeb695060084096b.queue_time_us) (tablet-download-pool-a924487225dc4246aeb695060084096b.run_cpu_time_us, tablet-download-pool-a924487225dc4246aeb695060084096b.run_cpu_time_us) (tablet-download-pool-a924487225dc4246aeb695060084096b.run_wall_time_us, tablet-download-pool-a924487225dc4246aeb695060084096b.run_wall_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.queue_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.queue_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_cpu_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_cpu_time_us) (tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_wall_time_us, tablet-download-pool-c917c5849f54411eb55d69ab33456bf4.run_wall_time_us) (tablet-download-pool-d44b2c67014742af9a65a3576f509702.queue_time_us, tablet-download-pool-d44b2c67014742af9a65a3576f509702.queue_time_us) (tablet-download-pool-d44b2c67014742af9a65a3576f509702.run_cpu_time_us, tablet-download-pool-d44b2c67014742af9a65a3576f509702.run_cpu_time_us) (tablet-download-pool-d44b2c67014742af9a65a3576f509702.run_wall_time_us, tablet-download-pool-d44b2c67014742af9a65a3576f509702.run_wall_time_us) (tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.queue_time_us, tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.queue_time_us) (tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.run_cpu_time_us, tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.run_cpu_time_us) (tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.run_wall_time_us, tablet-download-pool-d9a2990625964c0e8b79780890b0e6c3.run_wall_time_us) (tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.queue_time_us, tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.queue_time_us) (tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.run_cpu_time_us, tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.run_cpu_time_us) (tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.run_wall_time_us, tablet-download-pool-dcdde0a31f7d4fd0bf20c894f215e3b6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fd4a92cfdcd google::LogMessage::Fail() at ??:0
@ 0x7fd4a92d3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd4a92cf7cc google::LogMessage::Flush() at ??:0
@ 0x7fd4a92d0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd4aa739044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd4aa71df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd4aa71befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fd4ad2ce77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fd4ad2d7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fd4ad35876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fd4ad36c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fd4ad357226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fd4ad366c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c454a9b75c std::function<>::operator()() at ??:0
@ 0x7fd4aa720e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd4aa7216ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd4aa72303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c454a9b75c std::function<>::operator()() at ??:0
@ 0x7fd4aa710807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd4aaf4e6db start_thread at ??:0
@ 0x7fd4a848671f clone at ??:0
I20240503 08:05:26.935907 13076 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:05:26.937709 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:26.938367 15281 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:26.940436 15281 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.944173 12668 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.945037 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } }
I20240503 08:05:26.947626 14768 tablet_service.cc:1514] Processing DeleteTablet for tablet a45748a98c3447d3aa2e8de94981a745 with delete_type TABLET_DATA_TOMBSTONED (TS b052e89b63c24258872c832a6a0d4d4d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49158
I20240503 08:05:26.948137 13076 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
W20240503 08:05:26.949949 13010 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): Couldn't send request to peer af712db2f85249adbe718e8ba0549f84. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.950177 12946 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:26.950517 15281 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.952169 15281 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.954363 12946 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.958066 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a45748a98c3447d3aa2e8de94981a745 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:26.958637 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.959460 12670 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 8 to 9, VOTER af712db2f85249adbe718e8ba0549f84 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:26.960464 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:26.961894 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:26.967439 17500 tablet_replica.cc:330] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: stopping tablet replica
I20240503 08:05:26.967592 17500 raft_consensus.cc:2229] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:26.970873 17500 raft_consensus.cc:2258] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:26.975100 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:26.975443 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:26.975723 17500 ts_tablet_manager.cc:1875] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:26.983321 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:26.978907 17500 ts_tablet_manager.cc:1888] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:05:26.998890 17500 log.cc:1196] T a45748a98c3447d3aa2e8de94981a745 P b052e89b63c24258872c832a6a0d4d4d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a45748a98c3447d3aa2e8de94981a745
I20240503 08:05:26.999641 12659 catalog_manager.cc:4830] TS b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): tablet a45748a98c3447d3aa2e8de94981a745 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
I20240503 08:05:27.133280 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.135818 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.156034 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.174168 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.175503 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.183888 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.184368 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:27.197894 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.205323 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.244419 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.253746 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:27.268576 13012 connection.cc:535] client connection to 127.0.99.133:38311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:27.268594 14724 connection.cc:535] client connection to 127.0.99.133:38311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:27.268911 12750 connection.cc:535] client connection to 127.0.99.133:38311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:27.269011 14593 connection.cc:535] client connection to 127.0.99.133:38311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:27.269085 14593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 9 similar messages]
W20240503 08:05:27.269286 12881 connection.cc:535] client connection to 127.0.99.133:38311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:27.271986 13143 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:27.273140 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:27.283229 12750 consensus_peers.cc:485] T 389b40c2136d45ab86ccf49df8d1b71e P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.283352 12750 consensus_peers.cc:485] T dcdde0a31f7d4fd0bf20c894f215e3b6 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.283943 12750 consensus_peers.cc:485] T a924487225dc4246aeb695060084096b P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.294040 14593 consensus_peers.cc:485] T d9a2990625964c0e8b79780890b0e6c3 P 8398e6f89bf5419d81ed5bed991da0cf -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.295150 14789 raft_consensus.cc:1203] T 047aba81fad543ce8b562a184746b163 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240503 08:05:27.300474 13012 consensus_peers.cc:485] T a60b868dcc504bb29d08e2752c8dd44d P 39827620daed48cb8f137442e1f1da76 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.300869 14724 consensus_peers.cc:485] T 5cee7c94dd0941d99d9dd6ceb39fc190 P b052e89b63c24258872c832a6a0d4d4d -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.300932 14724 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.304589 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.306555 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.317337 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:05:27.318199 12881 consensus_peers.cc:485] T c917c5849f54411eb55d69ab33456bf4 P 0cf63591cfc244028e1fc368c2bca079 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:27.322544 13143 consensus_peers.cc:485] T d44b2c67014742af9a65a3576f509702 P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.336318 17683 ts_tablet_manager.cc:902] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Initiating tablet copy from peer b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:05:27.336956 17683 tablet_copy_client.cc:308] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37289
I20240503 08:05:27.337764 14808 tablet_copy_service.cc:140] P b052e89b63c24258872c832a6a0d4d4d: Received BeginTabletCopySession request for tablet 48e4f2a993164fb6b1df674311698006 from peer 39827620daed48cb8f137442e1f1da76 ({username='slave'} at 127.0.99.131:56735)
I20240503 08:05:27.337867 14808 tablet_copy_service.cc:161] P b052e89b63c24258872c832a6a0d4d4d: Beginning new tablet copy session on tablet 48e4f2a993164fb6b1df674311698006 from peer 39827620daed48cb8f137442e1f1da76 at {username='slave'} at 127.0.99.131:56735: session id = 39827620daed48cb8f137442e1f1da76-48e4f2a993164fb6b1df674311698006
I20240503 08:05:27.339082 14808 tablet_copy_source_session.cc:205] T 48e4f2a993164fb6b1df674311698006 P b052e89b63c24258872c832a6a0d4d4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:27.340288 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4f2a993164fb6b1df674311698006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:27.347807 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.349862 14970 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:10.656196 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.658329 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.667338 14977 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:11.639847 14977 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:12.104921 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.105118 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.106747 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.106844 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.106897 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.106987 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.107029 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.107111 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.107148 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.107196 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.107225 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.107291 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.107322 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.107348 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.107374 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:12.210822 14977 rebalancer_tool.cc:1074] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.302724 14977 rebalancer_tool.cc:1084] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> 'b052e89b63c24258872c832a6a0d4d4d' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 08:00:12.302829 14977 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:12.568313 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.568385 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.568444 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.568538 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.568588 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.568615 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.568647 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.568675 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.568718 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.568748 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.568778 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.568814 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.568840 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.568861 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.568888 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.568918 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.568961 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.569212 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:12.619287 14977 rebalancer_tool.cc:1074] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.695861 14977 rebalancer_tool.cc:1074] tablet d9a2990625964c0e8b79780890b0e6c3: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.745891 14977 rebalancer_tool.cc:1084] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move ignored: Illegal state: tablet d44b2c67014742af9a65a3576f509702: replica 82cd33f2bae4419b946095f8eb93e1e4 is already present
I20240503 08:00:12.745997 14977 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:13.371402 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.373020 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.379024 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.379127 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.379175 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.379237 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.408072 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.408185 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.408371 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.409147 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.410215 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.410292 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.410343 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.410409 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.410454 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.410493 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.410535 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.410578 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.410617 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.410658 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.410699 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.503787 14977 rebalancer_tool.cc:1074] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:13.669536 14977 rebalancer_tool.cc:1074] tablet 5480e544dca3428ab4fcf190577d5331: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:13.754629 14977 rebalancer_tool.cc:1074] tablet d376ac1ea05c482ca8f03ddf0f5fcd47: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:13.807595 14977 rebalancer_tool.cc:1074] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:13.923563 14977 rebalancer_tool.cc:1084] tablet f1151731adc9407b86d143d6d41f69b4: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 08:00:13.923645 14977 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:14.359073 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:14.359221 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.359338 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:14.359397 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.359448 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.359498 14977 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.377760 14977 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:14.458680 14977 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:14.551007 14977 rebalancer_tool.cc:1074] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:14.561686 14977 rebalancer_tool.cc:1074] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:15.709292 14977 rebalancer_tool.cc:1123] tablet 7c630e5cf7c0419cbf5d13413d22d317: '39827620daed48cb8f137442e1f1da76' -> '' move is abandoned: Remote error: Service unavailable: leader transfer in progress
I20240503 08:00:18.961545 14977 rebalancer_tool.cc:1136] tablet d9a2990625964c0e8b79780890b0e6c3: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:18.984088 14977 rebalancer_tool.cc:1136] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:19.041702 14977 rebalancer_tool.cc:1136] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:20.136400 14977 rebalancer_tool.cc:1074] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:20.709069 14977 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:20.724042 14977 rebalancer_tool.cc:1136] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:21.369309 14977 rebalancer_tool.cc:1136] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:21.771158 14977 rebalancer_tool.cc:1136] tablet 5480e544dca3428ab4fcf190577d5331: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:21.790735 14977 rebalancer_tool.cc:1136] tablet d376ac1ea05c482ca8f03ddf0f5fcd47: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:21.920605 14977 rebalancer_tool.cc:1074] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:21.970145 14977 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:21.992702 14977 rebalancer_tool.cc:1074] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:22.022727 14977 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:22.835707 14977 rebalancer_tool.cc:1136] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:22.863670 14977 rebalancer_tool.cc:1136] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:22.877689 14977 rebalancer_tool.cc:1136] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:22.919214 14977 rebalancer_tool.cc:1136] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:23.133219 14977 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:23.155376 14977 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:23.170758 14977 rebalancer_tool.cc:1074] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:23.191542 14977 rebalancer_tool.cc:1136] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:25.412874 14977 rebalancer_tool.cc:1074] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:25.440598 14977 rebalancer_tool.cc:1136] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
W20240503 08:00:26.289376 23954 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:26.196940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:35323 (local address 127.0.0.1:54524)
0503 08:00:26.210143 (+ 13203us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:26.210156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:26.210416 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:26.288683 (+ 78267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":13145,"thread_start_us":57,"threads_started":1}
I20240503 08:00:26.340167 14977 rebalancer_tool.cc:1136] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:26.346514 14977 rebalancer_tool.cc:1074] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:26.353366 14977 rebalancer_tool.cc:1136] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:26.360018 14977 rebalancer_tool.cc:1136] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
W20240503 08:00:48.846089 2949 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.000s sys 0.000s
W20240503 08:01:53.987545 4481 meta_cache.cc:1097] Time spent processing table locations response: real 0.072s user 0.000s sys 0.000s
W20240503 08:01:53.987723 4481 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 72064us
W20240503 08:02:03.987763 9427 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 58733.3us
W20240503 08:02:28.100277 22042 meta_cache.cc:1097] Time spent processing table locations response: real 0.056s user 0.004s sys 0.000s
W20240503 08:02:28.100419 22042 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 56147.8us
W20240503 08:02:28.100288 22061 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.002s sys 0.000s
W20240503 08:02:28.107177 22060 meta_cache.cc:1261] Time spent looking up entry by key: real 0.061s user 0.003s sys 0.000s
W20240503 08:02:28.108058 22062 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.003s sys 0.000s
W20240503 08:02:28.108507 22063 meta_cache.cc:1261] Time spent looking up entry by key: real 0.063s user 0.000s sys 0.003s
W20240503 08:04:52.796810 31550 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ListTabletServers -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 50572.8us
I20240503 08:05:27.011579 14977 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
W20240503 08:05:27.270503 17531 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:05:27.167631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:38311 (local address 127.0.0.1:55334)
0503 08:05:27.189846 (+ 22215us) negotiation.cc:107] Waiting for socket to connect
0503 08:05:27.189858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:05:27.189920 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:05:27.270373 (+ 80453us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22157,"thread_start_us":77,"threads_started":1}
Illegal state: tablet server eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): unacceptable health status UNAVAILABLE
I20240503 08:05:27.353540 17683 tablet_copy_client.cc:783] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:27.358649 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.359340 14972 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:10.649974 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.650785 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.655660 14976 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 08:00:10.920123 15132 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.000s sys 0.000s
I20240503 08:00:11.642824 14976 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:12.066852 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.067109 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.067162 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.067193 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.067276 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.067317 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.067365 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.067418 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.067448 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.067479 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.067508 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.067538 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.067565 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.067593 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.067620 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:12.222179 14976 rebalancer_tool.cc:1074] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.271199 14976 rebalancer_tool.cc:1074] tablet 7d61aa445a50417b82916019c2a4298c: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.358022 14976 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.518580 14976 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.589265 14976 rebalancer_tool.cc:1074] tablet c917c5849f54411eb55d69ab33456bf4: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.641345 14976 rebalancer_tool.cc:1074] tablet a60b868dcc504bb29d08e2752c8dd44d: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.698452 14976 rebalancer_tool.cc:1074] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.821676 14976 rebalancer_tool.cc:1074] tablet d9a2990625964c0e8b79780890b0e6c3: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.859896 14976 rebalancer_tool.cc:1084] tablet 1c84f7a57e0945d9a25a70798f054bd1: '39827620daed48cb8f137442e1f1da76' -> 'b052e89b63c24258872c832a6a0d4d4d' move ignored: Illegal state: tablet 1c84f7a57e0945d9a25a70798f054bd1: replica b052e89b63c24258872c832a6a0d4d4d is already present
I20240503 08:00:12.859982 14976 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:13.424480 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.424551 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.424585 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.424649 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.424687 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.424724 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.442988 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.443084 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.443127 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.443161 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.443186 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.443221 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.443249 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.443276 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.443305 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.443337 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.443361 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:13.484318 14976 rebalancer_tool.cc:1074] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:13.706815 14976 rebalancer_tool.cc:1074] tablet b094d04434d24923806b0e9e209c3fca: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:13.726881 14976 rebalancer_tool.cc:1084] tablet 9e7f229898c24572815177628720521f: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move ignored: Illegal state: tablet 9e7f229898c24572815177628720521f: replica 8398e6f89bf5419d81ed5bed991da0cf is already present
I20240503 08:00:13.727393 14976 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:14.273371 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:14.273476 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:14.273602 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.273674 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.273739 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:14.273784 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.273849 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.273891 14976 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:14.372579 14976 rebalancer_tool.cc:1074] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:14.432737 14976 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:14.486088 14976 rebalancer_tool.cc:1074] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:14.537815 14976 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:17.664172 14976 rebalancer_tool.cc:1136] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:18.253479 14976 rebalancer_tool.cc:1136] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:18.917527 14976 rebalancer_tool.cc:1136] tablet a60b868dcc504bb29d08e2752c8dd44d: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:18.929486 14976 rebalancer_tool.cc:1136] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:21.160526 14976 rebalancer_tool.cc:1074] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:21.762737 14976 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:21.770902 14976 rebalancer_tool.cc:1136] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:21.808890 14976 rebalancer_tool.cc:1136] tablet 7c630e5cf7c0419cbf5d13413d22d317: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:21.902979 14976 rebalancer_tool.cc:1136] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:21.958472 14976 rebalancer_tool.cc:1136] tablet d9a2990625964c0e8b79780890b0e6c3: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:21.980849 14976 rebalancer_tool.cc:1136] tablet 7d61aa445a50417b82916019c2a4298c: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:21.997915 14976 rebalancer_tool.cc:1136] tablet b094d04434d24923806b0e9e209c3fca: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:22.030663 14976 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:22.080432 14976 rebalancer_tool.cc:1136] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:22.094566 14976 rebalancer_tool.cc:1136] tablet c917c5849f54411eb55d69ab33456bf4: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:22.129940 14976 rebalancer_tool.cc:1074] tablet 02d3e4066cd14aa18619310a4490246b: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:22.139526 14976 rebalancer_tool.cc:1074] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:22.147526 14976 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:22.159396 14976 rebalancer_tool.cc:1074] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:23.174440 14976 rebalancer_tool.cc:1136] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:24.838519 14976 rebalancer_tool.cc:1136] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
W20240503 08:00:26.289391 23981 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:26.043961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:35323 (local address 127.0.0.1:54510)
0503 08:00:26.054769 (+ 10808us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:26.054780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:26.055205 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:26.288683 (+233478us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":10756}
I20240503 08:00:26.560424 14976 rebalancer_tool.cc:1136] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:26.570816 14976 rebalancer_tool.cc:1136] tablet 02d3e4066cd14aa18619310a4490246b: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
W20240503 08:00:40.811275 31392 meta_cache.cc:1261] Time spent looking up entry by key: real 0.069s user 0.000s sys 0.000s
W20240503 08:00:40.834134 31388 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.000s sys 0.000s
W20240503 08:02:58.726383 4622 meta_cache.cc:1261] Time spent looking up entry by key: real 0.082s user 0.000s sys 0.000s
W20240503 08:02:58.737470 4624 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.000s sys 0.000s
I20240503 08:05:26.941987 14976 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
W20240503 08:05:27.270201 17612 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:05:27.124507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:38311 (local address 127.0.0.1:55322)
0503 08:05:27.125784 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0503 08:05:27.125796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:05:27.125861 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:05:27.270071 (+144210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":60,"threads_started":1}
Illegal state: tablet server eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): unacceptable health status UNAVAILABLE
I20240503 08:05:27.373373 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:27.382460 12750 consensus_peers.cc:485] T 5480e544dca3428ab4fcf190577d5331 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.384480 17683 tablet_copy_client.cc:647] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:27.388164 17683 tablet_copy_client.cc:523] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:27.390175 17683 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Bootstrap starting.
I20240503 08:05:27.400858 17683 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:27.401276 17683 tablet_bootstrap.cc:492] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Bootstrap complete.
I20240503 08:05:27.401552 17683 ts_tablet_manager.cc:1367] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.006s
I20240503 08:05:27.402088 17683 raft_consensus.cc:348] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:05:27.402338 17683 raft_consensus.cc:729] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Initialized, Role: FOLLOWER
I20240503 08:05:27.402536 17683 consensus_queue.cc:260] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } }
I20240503 08:05:27.403049 17683 ts_tablet_manager.cc:1398] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:27.403772 14808 tablet_copy_service.cc:342] P b052e89b63c24258872c832a6a0d4d4d: Request end of tablet copy session 39827620daed48cb8f137442e1f1da76-48e4f2a993164fb6b1df674311698006 received from {username='slave'} at 127.0.99.131:56735
I20240503 08:05:27.403842 14808 tablet_copy_service.cc:433] P b052e89b63c24258872c832a6a0d4d4d: ending tablet copy session 39827620daed48cb8f137442e1f1da76-48e4f2a993164fb6b1df674311698006 on tablet 48e4f2a993164fb6b1df674311698006 with peer 39827620daed48cb8f137442e1f1da76
I20240503 08:05:27.422354 14971 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:10.659057 14978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.659510 14978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.663705 14978 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:11.703217 14978 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:12.225968 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226199 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.226289 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226398 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226442 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.226481 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.226558 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.226599 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226640 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.226667 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226696 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.226732 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.226765 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.226819 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:12.288460 14978 rebalancer_tool.cc:1074] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.344408 14978 rebalancer_tool.cc:1074] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.558137 14978 rebalancer_tool.cc:1074] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.627346 14978 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.683512 14978 rebalancer_tool.cc:1074] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.836886 14978 rebalancer_tool.cc:1074] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.999696 14978 rebalancer_tool.cc:1074] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:13.046573 14978 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:13.561373 14978 rebalancer_tool.cc:1074] tablet 1c84f7a57e0945d9a25a70798f054bd1: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:13.680574 14978 rebalancer_tool.cc:1074] tablet a60b868dcc504bb29d08e2752c8dd44d: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:13.777637 14978 rebalancer_tool.cc:1074] tablet c917c5849f54411eb55d69ab33456bf4: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:13.942842 14978 rebalancer_tool.cc:1074] tablet 7d61aa445a50417b82916019c2a4298c: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:14.114017 14978 rebalancer_tool.cc:1074] tablet 5480e544dca3428ab4fcf190577d5331: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:14.403800 14978 rebalancer_tool.cc:1074] tablet 5212dff7ded143a7923bb6176466b0fa: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:14.543327 14978 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:14.570315 14978 rebalancer_tool.cc:1084] tablet 047aba81fad543ce8b562a184746b163: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move ignored: Illegal state: tablet 047aba81fad543ce8b562a184746b163: replica eae9f53215a34bc88abe7b3c1c223319 is already present
I20240503 08:00:14.570406 14978 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:15.081527 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:15.081715 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:15.081773 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:15.081817 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:15.081892 14978 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:15.091490 14978 rebalancer_tool.cc:1074] tablet d9a2990625964c0e8b79780890b0e6c3: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:15.101006 14978 rebalancer_tool.cc:1074] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:15.125245 14978 rebalancer_tool.cc:1074] tablet f1151731adc9407b86d143d6d41f69b4: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:15.160094 14978 rebalancer_tool.cc:1074] tablet 076781af2ac14ddc87d31e3ec77db08f: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:15.177729 14978 rebalancer_tool.cc:1074] tablet 9e7f229898c24572815177628720521f: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:17.660387 14978 rebalancer_tool.cc:1136] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:18.799558 14978 rebalancer_tool.cc:1136] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:18.854503 14978 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:20.021507 14978 rebalancer_tool.cc:1136] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:20.039547 14978 rebalancer_tool.cc:1136] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:20.049166 14978 rebalancer_tool.cc:1136] tablet 7d61aa445a50417b82916019c2a4298c: '6c358da9faef48c4adb12cce628aeb38' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:21.499311 14978 rebalancer_tool.cc:1136] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:21.524166 14978 rebalancer_tool.cc:1136] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:21.553292 14978 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:22.019361 14978 rebalancer_tool.cc:1123] tablet d9a2990625964c0e8b79780890b0e6c3: '6c358da9faef48c4adb12cce628aeb38' -> '' move is abandoned: Not found: Server with UUID 6c358da9faef48c4adb12cce628aeb38 not a member of the config. RaftConfig: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:22.043082 14978 rebalancer_tool.cc:1136] tablet 9e7f229898c24572815177628720521f: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:22.050945 14978 rebalancer_tool.cc:1136] tablet f1151731adc9407b86d143d6d41f69b4: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:22.077049 14978 rebalancer_tool.cc:1136] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:22.121405 14978 rebalancer_tool.cc:1136] tablet 076781af2ac14ddc87d31e3ec77db08f: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:22.136168 14978 rebalancer_tool.cc:1136] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:22.176602 14978 rebalancer_tool.cc:1136] tablet 5480e544dca3428ab4fcf190577d5331: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:22.189906 14978 rebalancer_tool.cc:1136] tablet a60b868dcc504bb29d08e2752c8dd44d: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:22.198997 14978 rebalancer_tool.cc:1136] tablet 5212dff7ded143a7923bb6176466b0fa: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:22.294420 14978 rebalancer_tool.cc:1136] tablet 1c84f7a57e0945d9a25a70798f054bd1: '39827620daed48cb8f137442e1f1da76' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:23.180274 14978 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:23.199347 14978 rebalancer_tool.cc:1074] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:23.211913 14978 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:23.222086 14978 rebalancer_tool.cc:1074] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:24.437355 14978 rebalancer_tool.cc:1136] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:24.452260 14978 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:24.953253 14978 rebalancer_tool.cc:1136] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:24.959858 14978 rebalancer_tool.cc:1136] tablet 7c630e5cf7c0419cbf5d13413d22d317: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:24.971150 14978 rebalancer_tool.cc:1136] tablet c917c5849f54411eb55d69ab33456bf4: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:25.419970 14978 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move scheduled
I20240503 08:00:25.597648 14978 rebalancer_tool.cc:1074] tablet 389b40c2136d45ab86ccf49df8d1b71e: 'b052e89b63c24258872c832a6a0d4d4d' -> '0cf63591cfc244028e1fc368c2bca079' move scheduled
I20240503 08:00:25.608666 14978 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '8398e6f89bf5419d81ed5bed991da0cf' -> '' move scheduled
I20240503 08:00:25.683502 14978 rebalancer_tool.cc:1074] tablet 1c84f7a57e0945d9a25a70798f054bd1: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:25.792840 14978 rebalancer_tool.cc:1074] tablet a45748a98c3447d3aa2e8de94981a745: 'b052e89b63c24258872c832a6a0d4d4d' -> '' move scheduled
I20240503 08:00:26.801976 14978 rebalancer_tool.cc:1074] tablet 70c5de226c7a4b349eef86a79d172851: 'eae9f53215a34bc88abe7b3c1c223319' -> '0cf63591cfc244028e1fc368c2bca079' move scheduled
I20240503 08:00:26.858206 14978 rebalancer_tool.cc:1074] tablet a60b868dcc504bb29d08e2752c8dd44d: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move scheduled
I20240503 08:00:31.354626 14978 rebalancer_tool.cc:1136] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '8398e6f89bf5419d81ed5bed991da0cf' -> '' move completed: OK
I20240503 08:00:31.366353 14978 rebalancer_tool.cc:1136] tablet 389b40c2136d45ab86ccf49df8d1b71e: 'b052e89b63c24258872c832a6a0d4d4d' -> '0cf63591cfc244028e1fc368c2bca079' move completed: OK
W20240503 08:01:52.311779 3696 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.000s sys 0.000s
W20240503 08:03:05.277827 7942 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.000s sys 0.000s
W20240503 08:03:17.887521 15311 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
W20240503 08:03:24.829257 18910 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.000s sys 0.000s
W20240503 08:03:59.384639 3791 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
W20240503 08:04:37.791777 23984 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.GetStatus -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 61604.3us
I20240503 08:05:25.926131 14978 rebalancer_tool.cc:1136] tablet a60b868dcc504bb29d08e2752c8dd44d: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move completed: OK
I20240503 08:05:26.682842 14978 rebalancer_tool.cc:1123] tablet 70c5de226c7a4b349eef86a79d172851: 'eae9f53215a34bc88abe7b3c1c223319' -> '0cf63591cfc244028e1fc368c2bca079' move is abandoned: Not found: Server with UUID eae9f53215a34bc88abe7b3c1c223319 not a member of the config. RaftConfig: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:27.017664 14978 rebalancer_tool.cc:1136] tablet 1c84f7a57e0945d9a25a70798f054bd1: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: Incomplete: tablet 1c84f7a57e0945d9a25a70798f054bd1, TS af712db2f85249adbe718e8ba0549f84 -> TS eae9f53215a34bc88abe7b3c1c223319 move failed, target replica disappeared
I20240503 08:05:27.031333 14978 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move completed: OK
I20240503 08:05:27.053273 14978 rebalancer_tool.cc:1136] tablet a45748a98c3447d3aa2e8de94981a745: 'b052e89b63c24258872c832a6a0d4d4d' -> '' move completed: OK
Illegal state: tablet server 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): unacceptable health status UNAVAILABLE
I20240503 08:05:27.433027 15586 raft_consensus.cc:998] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84: Attempting to remove follower eae9f53215a34bc88abe7b3c1c223319 from the Raft config. Reason: Leader has been unable to successfully communicate with peer eae9f53215a34bc88abe7b3c1c223319 for more than 300 seconds (300.279s)
I20240503 08:05:27.433353 15586 consensus_queue.cc:237] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:27.434526 12946 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:27.435142 12815 raft_consensus.cc:1261] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Refusing update from remote peer af712db2f85249adbe718e8ba0549f84: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:27.436832 17731 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:27.437222 17729 consensus_queue.cc:1035] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:27.438899 17729 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P af712db2f85249adbe718e8ba0549f84 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:27.439191 12946 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:27.441843 12668 catalog_manager.cc:5484] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 0cf63591cfc244028e1fc368c2bca079 reported cstate change: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "af712db2f85249adbe718e8ba0549f84" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:27.446861 12813 raft_consensus.cc:2932] T d376ac1ea05c482ca8f03ddf0f5fcd47 P 6c358da9faef48c4adb12cce628aeb38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
W20240503 08:05:27.453317 13143 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:27.455909 17734 ts_tablet_manager.cc:902] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Initiating tablet copy from peer 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153)
I20240503 08:05:27.456679 17734 tablet_copy_client.cc:308] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43153
I20240503 08:05:27.457110 13097 tablet_copy_service.cc:140] P 39827620daed48cb8f137442e1f1da76: Received BeginTabletCopySession request for tablet a45748a98c3447d3aa2e8de94981a745 from peer af712db2f85249adbe718e8ba0549f84 ({username='slave'} at 127.0.99.132:52341)
I20240503 08:05:27.457213 13097 tablet_copy_service.cc:161] P 39827620daed48cb8f137442e1f1da76: Beginning new tablet copy session on tablet a45748a98c3447d3aa2e8de94981a745 from peer af712db2f85249adbe718e8ba0549f84 at {username='slave'} at 127.0.99.132:52341: session id = af712db2f85249adbe718e8ba0549f84-a45748a98c3447d3aa2e8de94981a745
I20240503 08:05:27.458104 13097 tablet_copy_source_session.cc:205] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:27.459002 17734 ts_tablet_manager.cc:1875] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:27.461815 17734 ts_tablet_manager.cc:1888] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:27.462070 17734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45748a98c3447d3aa2e8de94981a745. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:27.464015 17734 tablet_copy_client.cc:783] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:27.468662 17734 tablet_copy_client.cc:647] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:27.473207 17734 tablet_copy_client.cc:523] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:27.474879 17734 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Bootstrap starting.
I20240503 08:05:27.484383 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.485098 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.486481 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.486614 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:27.486876 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.499518 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.509136 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.510658 17734 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:27.510879 17734 tablet_bootstrap.cc:492] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Bootstrap complete.
I20240503 08:05:27.512617 12793 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.514755 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.518606 17734 ts_tablet_manager.cc:1367] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.006s
I20240503 08:05:27.519170 17734 raft_consensus.cc:348] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:27.521030 17734 raft_consensus.cc:729] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af712db2f85249adbe718e8ba0549f84, State: Initialized, Role: FOLLOWER
I20240503 08:05:27.521239 17734 consensus_queue.cc:260] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:27.524911 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.532770 17734 ts_tablet_manager.cc:1398] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 08:05:27.534704 13097 tablet_copy_service.cc:342] P 39827620daed48cb8f137442e1f1da76: Request end of tablet copy session af712db2f85249adbe718e8ba0549f84-a45748a98c3447d3aa2e8de94981a745 received from {username='slave'} at 127.0.99.132:52341
I20240503 08:05:27.534778 13097 tablet_copy_service.cc:433] P 39827620daed48cb8f137442e1f1da76: ending tablet copy session af712db2f85249adbe718e8ba0549f84-a45748a98c3447d3aa2e8de94981a745 on tablet a45748a98c3447d3aa2e8de94981a745 with peer af712db2f85249adbe718e8ba0549f84
I20240503 08:05:27.559398 14768 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:27.561703 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:27.571035 14789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45"
dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49254
I20240503 08:05:27.571187 14789 raft_consensus.cc:595] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Received request to transfer leadership
I20240503 08:05:27.599258 14973 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:10.646584 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.646859 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.654045 14975 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:11.706952 14975 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:12.118172 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.118408 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.118495 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.118561 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.118616 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.118682 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.118734 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.118803 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.118856 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.119040 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.119118 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.119182 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.119239 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.119303 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:12.213375 14975 rebalancer_tool.cc:1084] tablet 20cc823a5ce34916894cfde0a6614bd1: '0cf63591cfc244028e1fc368c2bca079' -> '82cd33f2bae4419b946095f8eb93e1e4' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 08:00:12.213480 14975 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:12.472455 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.472538 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.472604 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.472648 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.472700 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.472750 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.472783 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.472812 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.472844 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.472882 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.472914 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.473033 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.473098 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.473129 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.507719 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.507782 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.507817 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.507851 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.507917 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:12.573431 14975 rebalancer_tool.cc:1074] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.653096 14975 rebalancer_tool.cc:1074] tablet 389b40c2136d45ab86ccf49df8d1b71e: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.705576 14975 rebalancer_tool.cc:1074] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.810864 14975 rebalancer_tool.cc:1074] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.954625 14975 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: '39827620daed48cb8f137442e1f1da76' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:13.155519 14975 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:13.578859 14975 rebalancer_tool.cc:1074] tablet 9e7f229898c24572815177628720521f: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:13.629845 14975 rebalancer_tool.cc:1084] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move ignored: Illegal state: tablet 672fc9981d074f7e8f2324ba0b06be57: replica eae9f53215a34bc88abe7b3c1c223319 is already present
I20240503 08:00:13.629925 14975 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:13.928839 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.928948 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.929046 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.929085 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.929131 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.929200 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.929286 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.929317 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.929358 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.929402 14975 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:13.956699 14975 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:13.975387 14975 rebalancer_tool.cc:1074] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:13.993851 14975 rebalancer_tool.cc:1074] tablet a924487225dc4246aeb695060084096b: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:14.120334 14975 rebalancer_tool.cc:1074] tablet 70c5de226c7a4b349eef86a79d172851: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:14.287011 14975 rebalancer_tool.cc:1074] tablet d8994c38d0f640e9bd5b5c322c603b9f: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:19.182118 14975 rebalancer_tool.cc:1136] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:19.865226 14975 rebalancer_tool.cc:1136] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:20.656391 14975 rebalancer_tool.cc:1136] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: 'af712db2f85249adbe718e8ba0549f84' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:20.719692 14975 rebalancer_tool.cc:1074] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:22.764233 14975 rebalancer_tool.cc:1136] tablet a924487225dc4246aeb695060084096b: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:23.315177 14975 rebalancer_tool.cc:1136] tablet 389b40c2136d45ab86ccf49df8d1b71e: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:23.355973 14975 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:23.366331 14975 rebalancer_tool.cc:1136] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:23.543819 14975 rebalancer_tool.cc:1136] tablet a45748a98c3447d3aa2e8de94981a745: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:23.586553 14975 rebalancer_tool.cc:1136] tablet 7c630e5cf7c0419cbf5d13413d22d317: '39827620daed48cb8f137442e1f1da76' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:23.613040 14975 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:23.624619 14975 rebalancer_tool.cc:1074] tablet 20cc823a5ce34916894cfde0a6614bd1: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:25.660686 14975 rebalancer_tool.cc:1136] tablet d8994c38d0f640e9bd5b5c322c603b9f: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:25.719475 14975 rebalancer_tool.cc:1136] tablet 70c5de226c7a4b349eef86a79d172851: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:25.793234 14975 rebalancer_tool.cc:1136] tablet 9e7f229898c24572815177628720521f: '0cf63591cfc244028e1fc368c2bca079' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:25.803383 14975 rebalancer_tool.cc:1074] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:25.815265 14975 rebalancer_tool.cc:1074] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:25.911815 14975 rebalancer_tool.cc:1136] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:25.938215 14975 rebalancer_tool.cc:1136] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
W20240503 08:00:26.289388 24033 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:26.222107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:35323 (local address 127.0.0.1:54528)
0503 08:00:26.237131 (+ 15024us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:26.237143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:26.237204 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:26.288684 (+ 51480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":14973}
I20240503 08:00:26.359042 14975 rebalancer_tool.cc:1136] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
W20240503 08:03:50.178522 31541 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 54994.9us
W20240503 08:04:06.661710 7369 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.000s sys 0.000s
I20240503 08:05:26.910090 14975 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
W20240503 08:05:27.269801 17489 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:05:26.934052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:38311 (local address 127.0.0.1:55298)
0503 08:05:26.934393 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0503 08:05:26.934406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:05:26.934485 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:05:27.269650 (+335165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":69,"threads_started":1}
I20240503 08:05:27.271960 14975 rebalancer_tool.cc:1123] tablet 20cc823a5ce34916894cfde0a6614bd1: 'af712db2f85249adbe718e8ba0549f84' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Illegal state: tablet server eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): unacceptable health status UNAVAILABLE
I20240503 08:05:27.614434 13124 maintenance_manager.cc:419] P 39827620daed48cb8f137442e1f1da76: Scheduling UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006): 4584 bytes on disk
I20240503 08:05:27.615242 13014 maintenance_manager.cc:643] P 39827620daed48cb8f137442e1f1da76: UndoDeltaBlockGCOp(48e4f2a993164fb6b1df674311698006) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 08:05:27.618288 17130 raft_consensus.cc:979] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: : Instructing follower 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:05:27.618376 17130 raft_consensus.cc:1067] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Signalling peer 6c358da9faef48c4adb12cce628aeb38 to start an election
I20240503 08:05:27.619793 12813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45"
dest_uuid: "6c358da9faef48c4adb12cce628aeb38"
from {username='slave'} at 127.0.99.135:45337
I20240503 08:05:27.619935 12813 raft_consensus.cc:482] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:05:27.619990 12813 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:05:27.621048 12813 raft_consensus.cc:504] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:27.621438 12813 leader_election.cc:288] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 3 election: Requested vote from peers af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969), b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289)
I20240503 08:05:27.621829 14789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "b052e89b63c24258872c832a6a0d4d4d"
I20240503 08:05:27.621877 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ac9fbe5dfd4a4e89912d7e947eca45" candidate_uuid: "6c358da9faef48c4adb12cce628aeb38" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "af712db2f85249adbe718e8ba0549f84"
I20240503 08:05:27.621969 14789 raft_consensus.cc:3032] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:05:27.621987 13208 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:05:27.622014 14789 raft_consensus.cc:729] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b052e89b63c24258872c832a6a0d4d4d, State: Running, Role: LEADER
I20240503 08:05:27.622170 14789 consensus_queue.cc:260] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:27.622308 14789 raft_consensus.cc:3037] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:05:27.622802 13208 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 3.
I20240503 08:05:27.623039 14789 raft_consensus.cc:2454] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c358da9faef48c4adb12cce628aeb38 in term 3.
I20240503 08:05:27.623114 12748 leader_election.cc:302] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c358da9faef48c4adb12cce628aeb38, af712db2f85249adbe718e8ba0549f84; no voters:
I20240503 08:05:27.623279 17048 raft_consensus.cc:2781] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:05:27.623420 17048 raft_consensus.cc:686] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 3 LEADER]: Becoming Leader. State: Replica: 6c358da9faef48c4adb12cce628aeb38, State: Running, Role: LEADER
I20240503 08:05:27.623562 17048 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:27.626041 12673 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 reported cstate change: term changed from 2 to 3, leader changed from b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) to 6c358da9faef48c4adb12cce628aeb38 (127.0.99.129). New cstate: current_term: 3 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:27.632581 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.632611 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:27.638275 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.639892 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:27.657094 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:27.665234 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.748031 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.748031 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.748553 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.748553 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:27.754302 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 08:05:27.761198 12657 catalog_manager.cc:4573] TS eae9f53215a34bc88abe7b3c1c223319 (127.0.99.136:35323): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d376ac1ea05c482ca8f03ddf0f5fcd47: Network error: Client connection negotiation failed: client connection to 127.0.99.136:35323: connect: Connection refused (error 111)
I20240503 08:05:27.765861 13255 maintenance_manager.cc:419] P af712db2f85249adbe718e8ba0549f84: Scheduling UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745): 4838 bytes on disk
I20240503 08:05:27.767256 13145 maintenance_manager.cc:643] P af712db2f85249adbe718e8ba0549f84: UndoDeltaBlockGCOp(a45748a98c3447d3aa2e8de94981a745) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:05:27.770534 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:27.858042 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.861222 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.863291 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:27.865373 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:05:27.865790 13143 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:27.873090 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:27.874138 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.947125 13076 raft_consensus.cc:1203] T 48e4f2a993164fb6b1df674311698006 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 08:05:27.955476 15586 raft_consensus.cc:482] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82cd33f2bae4419b946095f8eb93e1e4)
I20240503 08:05:27.955608 15586 raft_consensus.cc:504] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:27.956053 15586 leader_election.cc:288] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
I20240503 08:05:27.956504 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "af712db2f85249adbe718e8ba0549f84" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
W20240503 08:05:27.958120 13143 leader_election.cc:334] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:27.958209 13143 leader_election.cc:302] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af712db2f85249adbe718e8ba0549f84; no voters: 39827620daed48cb8f137442e1f1da76, 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:05:27.958423 15586 raft_consensus.cc:2726] T 20cc823a5ce34916894cfde0a6614bd1 P af712db2f85249adbe718e8ba0549f84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:05:27.970752 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:27.972405 13056 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:05:27.973450 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:27.980362 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:27.992451 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:27.994627 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:28.007184 13208 raft_consensus.cc:1203] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:05:28.052544 14789 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 3 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:05:28.053303 17048 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:28.056226 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:05:28.056710 17797 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:28.070736 16680 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82cd33f2bae4419b946095f8eb93e1e4)
I20240503 08:05:28.070852 16680 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:28.071348 16680 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39827620daed48cb8f137442e1f1da76 (127.0.99.131:43153), 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311)
W20240503 08:05:28.072364 12881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 08:05:28.072898 12881 leader_election.cc:334] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:28.075639 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "0cf63591cfc244028e1fc368c2bca079" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "39827620daed48cb8f137442e1f1da76" is_pre_election: true
I20240503 08:05:28.076032 12879 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079; no voters: 39827620daed48cb8f137442e1f1da76, 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:05:28.076176 16680 raft_consensus.cc:2726] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:05:28.091382 17494 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82cd33f2bae4419b946095f8eb93e1e4)
I20240503 08:05:28.091516 17494 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:28.091985 17494 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
I20240503 08:05:28.092383 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079" is_pre_election: true
I20240503 08:05:28.092527 12946 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 2.
I20240503 08:05:28.092823 13013 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters:
I20240503 08:05:28.093159 17505 raft_consensus.cc:2781] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:05:28.093236 17505 raft_consensus.cc:482] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 82cd33f2bae4419b946095f8eb93e1e4)
I20240503 08:05:28.093279 17505 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:05:28.094440 17505 raft_consensus.cc:504] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:28.094729 17505 leader_election.cc:288] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 election: Requested vote from peers 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311), 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937)
W20240503 08:05:28.095049 13012 leader_election.cc:334] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:28.095207 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c5de226c7a4b349eef86a79d172851" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "0cf63591cfc244028e1fc368c2bca079"
I20240503 08:05:28.095328 12946 raft_consensus.cc:3037] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:05:28.096164 13012 leader_election.cc:334] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:28.096338 12946 raft_consensus.cc:2454] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39827620daed48cb8f137442e1f1da76 in term 3.
I20240503 08:05:28.096689 13013 leader_election.cc:302] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cf63591cfc244028e1fc368c2bca079, 39827620daed48cb8f137442e1f1da76; no voters: 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:05:28.097888 17494 raft_consensus.cc:2781] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:05:28.097975 17494 raft_consensus.cc:686] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [term 3 LEADER]: Becoming Leader. State: Replica: 39827620daed48cb8f137442e1f1da76, State: Running, Role: LEADER
I20240503 08:05:28.098129 17494 consensus_queue.cc:237] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } }
I20240503 08:05:28.100548 12670 catalog_manager.cc:5484] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 reported cstate change: term changed from 2 to 3, leader changed from 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133) to 39827620daed48cb8f137442e1f1da76 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } }
I20240503 08:05:28.101332 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:28.103085 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:28.103793 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:28.109077 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:28.121564 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:28.125619 13056 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:05:28.167436 12813 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } }
I20240503 08:05:28.168470 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 08:05:28.168831 17797 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:05:28.170580 17048 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:28.171033 13208 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:28.174077 12670 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 10 to 12, VOTER b052e89b63c24258872c832a6a0d4d4d (127.0.99.135) evicted. New cstate: current_term: 3 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } }
I20240503 08:05:28.176471 14768 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with delete_type TABLET_DATA_TOMBSTONED (TS b052e89b63c24258872c832a6a0d4d4d not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49158
I20240503 08:05:28.176838 18007 tablet_replica.cc:330] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: stopping tablet replica
I20240503 08:05:28.176971 18007 raft_consensus.cc:2229] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:28.177069 18007 raft_consensus.cc:2258] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:28.177536 12813 consensus_queue.cc:237] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:28.177712 18007 ts_tablet_manager.cc:1875] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:28.178726 13208 raft_consensus.cc:1261] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Refusing update from remote peer 6c358da9faef48c4adb12cce628aeb38: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
W20240503 08:05:28.178936 12749 consensus_peers.cc:485] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ac9fbe5dfd4a4e89912d7e947eca45. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:28.179258 17797 consensus_queue.cc:1035] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:05:28.180683 18007 ts_tablet_manager.cc:1888] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 08:05:28.180761 18007 log.cc:1196] T 72ac9fbe5dfd4a4e89912d7e947eca45 P b052e89b63c24258872c832a6a0d4d4d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/72ac9fbe5dfd4a4e89912d7e947eca45
I20240503 08:05:28.180750 17048 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P 6c358da9faef48c4adb12cce628aeb38 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.181113 13208 raft_consensus.cc:2932] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.181320 12659 catalog_manager.cc:4830] TS b052e89b63c24258872c832a6a0d4d4d (127.0.99.135:37289): tablet 72ac9fbe5dfd4a4e89912d7e947eca45 (table rebalance_test_table_4 [id=90a1130c18c1426080ffb6220d8e5846]) successfully deleted
I20240503 08:05:28.182566 12659 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 72ac9fbe5dfd4a4e89912d7e947eca45 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:28.183174 12671 catalog_manager.cc:5484] T 72ac9fbe5dfd4a4e89912d7e947eca45 P af712db2f85249adbe718e8ba0549f84 reported cstate change: config changed from index 12 to 13, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 3 leader_uuid: "6c358da9faef48c4adb12cce628aeb38" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "6c358da9faef48c4adb12cce628aeb38" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43847 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.219390 17505 raft_consensus.cc:482] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82cd33f2bae4419b946095f8eb93e1e4)
I20240503 08:05:28.219499 17505 raft_consensus.cc:504] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:28.219849 17505 leader_election.cc:288] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311), af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969)
I20240503 08:05:28.220353 13208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cc823a5ce34916894cfde0a6614bd1" candidate_uuid: "39827620daed48cb8f137442e1f1da76" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "af712db2f85249adbe718e8ba0549f84" is_pre_election: true
W20240503 08:05:28.220826 13012 leader_election.cc:334] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:28.220896 13012 leader_election.cc:302] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39827620daed48cb8f137442e1f1da76; no voters: 82cd33f2bae4419b946095f8eb93e1e4, af712db2f85249adbe718e8ba0549f84
I20240503 08:05:28.221015 17505 raft_consensus.cc:2726] T 20cc823a5ce34916894cfde0a6614bd1 P 39827620daed48cb8f137442e1f1da76 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:28.261353 13143 consensus_peers.cc:462] T 0e0d84a47cc34af9bdd6b955feb2c3ba P af712db2f85249adbe718e8ba0549f84 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111)
I20240503 08:05:28.437759 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:28.438786 13056 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:05:28.444236 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:28.444833 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:28.447986 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:28.454078 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:28.500486 13076 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } }
I20240503 08:05:28.501698 13208 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:05:28.502054 17505 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:28.503780 17494 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:28.504192 13208 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:28.506364 12665 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 9 to 10, VOTER 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } }
I20240503 08:05:28.509965 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet a45748a98c3447d3aa2e8de94981a745 with delete_type TABLET_DATA_TOMBSTONED (TS 0cf63591cfc244028e1fc368c2bca079 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53588
I20240503 08:05:28.510285 13076 consensus_queue.cc:237] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } }
I20240503 08:05:28.511250 18077 tablet_replica.cc:330] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: stopping tablet replica
I20240503 08:05:28.511447 18077 raft_consensus.cc:2229] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:28.511662 18077 raft_consensus.cc:2258] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:28.511713 13208 raft_consensus.cc:1261] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:28.512279 18077 ts_tablet_manager.cc:1875] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:28.512276 17505 consensus_queue.cc:1035] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:05:28.512451 13011 consensus_peers.cc:485] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 -> Peer 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134:39483): Couldn't send request to peer 8398e6f89bf5419d81ed5bed991da0cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a45748a98c3447d3aa2e8de94981a745. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:28.513914 17505 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.515945 12657 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a45748a98c3447d3aa2e8de94981a745 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 08:05:28.516510 12665 catalog_manager.cc:5484] T a45748a98c3447d3aa2e8de94981a745 P 39827620daed48cb8f137442e1f1da76 reported cstate change: config changed from index 10 to 11, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "39827620daed48cb8f137442e1f1da76" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.516649 13208 raft_consensus.cc:2932] T a45748a98c3447d3aa2e8de94981a745 P af712db2f85249adbe718e8ba0549f84 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8398e6f89bf5419d81ed5bed991da0cf (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "39827620daed48cb8f137442e1f1da76" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43153 } } peers { permanent_uuid: "af712db2f85249adbe718e8ba0549f84" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37969 } } peers { permanent_uuid: "8398e6f89bf5419d81ed5bed991da0cf" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39483 } } }
I20240503 08:05:28.517287 18077 ts_tablet_manager.cc:1888] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:05:28.517362 18077 log.cc:1196] T a45748a98c3447d3aa2e8de94981a745 P 0cf63591cfc244028e1fc368c2bca079: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a45748a98c3447d3aa2e8de94981a745
I20240503 08:05:28.518002 12660 catalog_manager.cc:4830] TS 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): tablet a45748a98c3447d3aa2e8de94981a745 (table rebalance_test_table_7 [id=5c041949c87644a9b4f1cf6b52a397f7]) successfully deleted
W20240503 08:05:28.549296 13012 consensus_peers.cc:485] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 -> Peer 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): Couldn't send request to peer 82cd33f2bae4419b946095f8eb93e1e4. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:28.582510 12946 raft_consensus.cc:1261] T 70c5de226c7a4b349eef86a79d172851 P 0cf63591cfc244028e1fc368c2bca079 [term 3 FOLLOWER]: Refusing update from remote peer 39827620daed48cb8f137442e1f1da76: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:05:28.583129 17494 consensus_queue.cc:1035] T 70c5de226c7a4b349eef86a79d172851 P 39827620daed48cb8f137442e1f1da76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:28.586534 12924 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:28.587970 12793 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:28.596853 14637 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:28.597546 13056 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:05:28.600786 13188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:28.614521 14768 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:05:28.645471 14969 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:10.654556 14974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:10.654799 14974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:10.663779 14974 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:11.638458 14974 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:12.121686 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.122215 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.122371 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.122457 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.122511 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.122572 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.122679 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.122750 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.124141 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.124255 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:12.124348 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.124416 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:12.124492 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:12.124548 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:12.124603 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 82cd33f2bae4419b946095f8eb93e1e4
I20240503 08:00:12.272980 14974 rebalancer_tool.cc:1074] tablet 9c7bf40aa0614923a19feed8bab430fa: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.414155 14974 rebalancer_tool.cc:1074] tablet a924487225dc4246aeb695060084096b: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.575227 14974 rebalancer_tool.cc:1074] tablet a60b868dcc504bb29d08e2752c8dd44d: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:12.648794 14974 rebalancer_tool.cc:1074] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:12.731076 14974 rebalancer_tool.cc:1074] tablet 1c84f7a57e0945d9a25a70798f054bd1: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:12.880649 14974 rebalancer_tool.cc:1074] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move scheduled
I20240503 08:00:12.999994 14974 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:13.014873 14974 rebalancer_tool.cc:1084] tablet 20cc823a5ce34916894cfde0a6614bd1: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move ignored: Illegal state: tablet 20cc823a5ce34916894cfde0a6614bd1: replica eae9f53215a34bc88abe7b3c1c223319 is already present
I20240503 08:00:13.014984 14974 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:13.544611 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.546605 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.546710 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.546759 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.546803 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server b052e89b63c24258872c832a6a0d4d4d to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.546852 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.546900 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.546989 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.547040 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.547088 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.547132 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ede3e1269b3418f8ad902769287e993: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.547214 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f8dc7f8c13f4d338c817f912734bd17: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.547268 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:13.547318 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:13.547361 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server eae9f53215a34bc88abe7b3c1c223319
W20240503 08:00:13.547417 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 90a1130c18c1426080ffb6220d8e5846: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:13.548168 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server b052e89b63c24258872c832a6a0d4d4d
I20240503 08:00:13.584156 14974 rebalancer_tool.cc:1074] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:13.731556 14974 rebalancer_tool.cc:1074] tablet 047aba81fad543ce8b562a184746b163: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:13.743839 14974 rebalancer_tool.cc:1074] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:13.844050 14974 rebalancer_tool.cc:1074] tablet f1151731adc9407b86d143d6d41f69b4: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move scheduled
I20240503 08:00:13.880645 14974 rebalancer_tool.cc:1084] tablet 5480e544dca3428ab4fcf190577d5331: '6c358da9faef48c4adb12cce628aeb38' -> '82cd33f2bae4419b946095f8eb93e1e4' move ignored: Illegal state: tablet 5480e544dca3428ab4fcf190577d5331: replica 82cd33f2bae4419b946095f8eb93e1e4 is already present
I20240503 08:00:13.880728 14974 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:14.315064 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:14.315212 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e36f346a18884285a63bee81e275ca47: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.315310 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0a7085823ec47c59ba5179f5a70d9e8: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.315388 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecd7ad5de8d5489baabee552c3a8e2ae: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.315444 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15283cbbb93542699834e6ff89a813c5: could not find any suitable replica to move from server 0cf63591cfc244028e1fc368c2bca079 to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:14.315481 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server 39827620daed48cb8f137442e1f1da76 to server 8398e6f89bf5419d81ed5bed991da0cf
W20240503 08:00:14.315534 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server af712db2f85249adbe718e8ba0549f84 to server 82cd33f2bae4419b946095f8eb93e1e4
W20240503 08:00:14.315618 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4aa669a3d80c4dc492176981a1978274: could not find any suitable replica to move from server 6c358da9faef48c4adb12cce628aeb38 to server eae9f53215a34bc88abe7b3c1c223319
I20240503 08:00:14.355919 14974 rebalancer_tool.cc:1074] tablet d9a2990625964c0e8b79780890b0e6c3: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:14.389441 14974 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:14.533645 14974 rebalancer_tool.cc:1074] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:16.753269 14974 rebalancer_tool.cc:1136] tablet f1151731adc9407b86d143d6d41f69b4: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:17.600065 14974 rebalancer_tool.cc:1136] tablet dcdde0a31f7d4fd0bf20c894f215e3b6: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:17.606643 14974 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: '6c358da9faef48c4adb12cce628aeb38' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:17.612974 14974 rebalancer_tool.cc:1136] tablet a924487225dc4246aeb695060084096b: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:18.309262 14974 rebalancer_tool.cc:1136] tablet 6ab3fc1d0e6a42f38406bb75f69b8cbc: 'af712db2f85249adbe718e8ba0549f84' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:00:20.716554 14974 rebalancer_tool.cc:1136] tablet 389b40c2136d45ab86ccf49df8d1b71e: '39827620daed48cb8f137442e1f1da76' -> '82cd33f2bae4419b946095f8eb93e1e4' move completed: OK
I20240503 08:00:20.733935 14974 rebalancer_tool.cc:1074] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:20.748538 14974 rebalancer_tool.cc:1136] tablet 70c5de226c7a4b349eef86a79d172851: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:20.762696 14974 rebalancer_tool.cc:1136] tablet d44b2c67014742af9a65a3576f509702: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:20.842903 14974 rebalancer_tool.cc:1136] tablet 7c630e5cf7c0419cbf5d13413d22d317: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:21.714329 14974 rebalancer_tool.cc:1136] tablet d8994c38d0f640e9bd5b5c322c603b9f: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:21.735734 14974 rebalancer_tool.cc:1136] tablet a60b868dcc504bb29d08e2752c8dd44d: '6c358da9faef48c4adb12cce628aeb38' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:21.775517 14974 rebalancer_tool.cc:1136] tablet 9c7bf40aa0614923a19feed8bab430fa: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:21.789217 14974 rebalancer_tool.cc:1136] tablet 1c84f7a57e0945d9a25a70798f054bd1: '0cf63591cfc244028e1fc368c2bca079' -> 'b052e89b63c24258872c832a6a0d4d4d' move completed: OK
I20240503 08:00:22.182709 14974 rebalancer_tool.cc:1074] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move scheduled
I20240503 08:00:22.193432 14974 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:22.291735 14974 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: 'af712db2f85249adbe718e8ba0549f84' -> '' move scheduled
I20240503 08:00:22.302598 14974 rebalancer_tool.cc:1074] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> '' move scheduled
I20240503 08:00:22.383185 14974 rebalancer_tool.cc:1136] tablet 5212dff7ded143a7923bb6176466b0fa: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:22.875001 14974 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:22.888470 14974 rebalancer_tool.cc:1136] tablet 02d3e4066cd14aa18619310a4490246b: '39827620daed48cb8f137442e1f1da76' -> '' move completed: OK
I20240503 08:00:23.013111 14974 rebalancer_tool.cc:1136] tablet d9a2990625964c0e8b79780890b0e6c3: '6c358da9faef48c4adb12cce628aeb38' -> '' move completed: OK
I20240503 08:00:23.172319 14974 rebalancer_tool.cc:1136] tablet 047aba81fad543ce8b562a184746b163: 'af712db2f85249adbe718e8ba0549f84' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: OK
I20240503 08:00:23.195093 14974 rebalancer_tool.cc:1074] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:23.209054 14974 rebalancer_tool.cc:1136] tablet 076781af2ac14ddc87d31e3ec77db08f: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
I20240503 08:00:23.217897 14974 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: 'af712db2f85249adbe718e8ba0549f84' -> '' move completed: OK
I20240503 08:00:23.647387 14974 rebalancer_tool.cc:1074] tablet 672fc9981d074f7e8f2324ba0b06be57: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move scheduled
I20240503 08:00:23.707199 14974 rebalancer_tool.cc:1074] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '0cf63591cfc244028e1fc368c2bca079' move scheduled
I20240503 08:00:25.080729 14974 rebalancer_tool.cc:1074] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: 'b052e89b63c24258872c832a6a0d4d4d' -> '6c358da9faef48c4adb12cce628aeb38' move scheduled
I20240503 08:00:25.372962 14974 rebalancer_tool.cc:1074] tablet 7c630e5cf7c0419cbf5d13413d22d317: '82cd33f2bae4419b946095f8eb93e1e4' -> 'af712db2f85249adbe718e8ba0549f84' move scheduled
I20240503 08:00:25.442268 14974 rebalancer_tool.cc:1074] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '8398e6f89bf5419d81ed5bed991da0cf' -> 'eae9f53215a34bc88abe7b3c1c223319' move scheduled
I20240503 08:00:25.542562 14974 rebalancer_tool.cc:1074] tablet a45748a98c3447d3aa2e8de94981a745: '0cf63591cfc244028e1fc368c2bca079' -> '' move scheduled
I20240503 08:00:25.598567 14974 rebalancer_tool.cc:1084] tablet 389b40c2136d45ab86ccf49df8d1b71e: 'b052e89b63c24258872c832a6a0d4d4d' -> '0cf63591cfc244028e1fc368c2bca079' move ignored: Illegal state: tablet 389b40c2136d45ab86ccf49df8d1b71e: replica 0cf63591cfc244028e1fc368c2bca079 is already present
I20240503 08:00:25.598635 14974 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:25.947072 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc180bf9c7ae466f9b724fe299385926: could not find any suitable replica to move from server 8398e6f89bf5419d81ed5bed991da0cf to server b052e89b63c24258872c832a6a0d4d4d
W20240503 08:00:25.954696 14974 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5c041949c87644a9b4f1cf6b52a397f7: could not find any suitable replica to move from server b052e89b63c24258872c832a6a0d4d4d to server 8398e6f89bf5419d81ed5bed991da0cf
I20240503 08:00:27.008253 14974 rebalancer_tool.cc:1074] tablet 48e4f2a993164fb6b1df674311698006: 'eae9f53215a34bc88abe7b3c1c223319' -> '8398e6f89bf5419d81ed5bed991da0cf' move scheduled
I20240503 08:00:28.221242 14974 rebalancer_tool.cc:1074] tablet d376ac1ea05c482ca8f03ddf0f5fcd47: 'eae9f53215a34bc88abe7b3c1c223319' -> '0cf63591cfc244028e1fc368c2bca079' move scheduled
I20240503 08:00:28.285532 14974 rebalancer_tool.cc:1074] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '8398e6f89bf5419d81ed5bed991da0cf' -> '' move scheduled
I20240503 08:00:28.820382 14974 rebalancer_tool.cc:1123] tablet 5cee7c94dd0941d99d9dd6ceb39fc190: '8398e6f89bf5419d81ed5bed991da0cf' -> '' move is abandoned: Not found: Server with UUID 8398e6f89bf5419d81ed5bed991da0cf not a member of the config. RaftConfig: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0cf63591cfc244028e1fc368c2bca079" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33937 } } peers { permanent_uuid: "b052e89b63c24258872c832a6a0d4d4d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37289 } } peers { permanent_uuid: "82cd33f2bae4419b946095f8eb93e1e4" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38311 } }
I20240503 08:00:31.316147 14974 rebalancer_tool.cc:1136] tablet 9c7bf40aa0614923a19feed8bab430fa: 'af712db2f85249adbe718e8ba0549f84' -> '0cf63591cfc244028e1fc368c2bca079' move completed: OK
I20240503 08:00:32.562289 14974 rebalancer_tool.cc:1136] tablet 7c630e5cf7c0419cbf5d13413d22d317: '82cd33f2bae4419b946095f8eb93e1e4' -> 'af712db2f85249adbe718e8ba0549f84' move completed: OK
W20240503 08:02:21.341100 18632 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 48241.1us
W20240503 08:02:33.388979 24655 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.0.99.190:37099, user_credentials={real_user=slave}} blocked reactor thread for 101394us
I20240503 08:05:26.246673 14974 rebalancer_tool.cc:1136] tablet 672fc9981d074f7e8f2324ba0b06be57: 'eae9f53215a34bc88abe7b3c1c223319' -> '' move completed: OK
I20240503 08:05:26.772938 14974 rebalancer_tool.cc:1136] tablet 48e4f2a993164fb6b1df674311698006: 'eae9f53215a34bc88abe7b3c1c223319' -> '8398e6f89bf5419d81ed5bed991da0cf' move completed: OK
I20240503 08:05:26.780092 14974 rebalancer_tool.cc:1136] tablet 0e0d84a47cc34af9bdd6b955feb2c3ba: '8398e6f89bf5419d81ed5bed991da0cf' -> 'eae9f53215a34bc88abe7b3c1c223319' move completed: Incomplete: tablet 0e0d84a47cc34af9bdd6b955feb2c3ba, TS 8398e6f89bf5419d81ed5bed991da0cf -> TS eae9f53215a34bc88abe7b3c1c223319 move failed, target replica disappeared
W20240503 08:05:27.270900 17493 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:05:26.936028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:38311 (local address 127.0.0.1:55308)
0503 08:05:26.939824 (+ 3796us) negotiation.cc:107] Waiting for socket to connect
0503 08:05:26.939835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:05:26.939898 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:05:27.270032 (+330134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:38311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3741,"thread_start_us":53,"threads_started":1}
I20240503 08:05:27.687736 14974 rebalancer_tool.cc:1136] tablet d376ac1ea05c482ca8f03ddf0f5fcd47: 'eae9f53215a34bc88abe7b3c1c223319' -> '0cf63591cfc244028e1fc368c2bca079' move completed: OK
I20240503 08:05:28.513021 14974 rebalancer_tool.cc:1136] tablet 72ac9fbe5dfd4a4e89912d7e947eca45: 'b052e89b63c24258872c832a6a0d4d4d' -> '6c358da9faef48c4adb12cce628aeb38' move completed: OK
I20240503 08:05:28.522257 14974 rebalancer_tool.cc:1136] tablet a45748a98c3447d3aa2e8de94981a745: '0cf63591cfc244028e1fc368c2bca079' -> '' move completed: OK
Illegal state: tablet server 82cd33f2bae4419b946095f8eb93e1e4 (127.0.99.133:38311): unacceptable health status UNAVAILABLE
E20240503 08:05:28.645838 398 external_mini_cluster.cc:943] Process with UUID 82cd33f2bae4419b946095f8eb93e1e4 has crashed
E20240503 08:05:28.645917 398 external_mini_cluster.cc:943] Process with UUID eae9f53215a34bc88abe7b3c1c223319 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 08:05:28.646596 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 12733
I20240503 08:05:28.654322 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 12864
W20240503 08:05:28.662180 13013 connection.cc:535] client connection to 127.0.99.130:33937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:05:28.662403 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 12995
W20240503 08:05:28.664862 14594 consensus_peers.cc:485] T 02d3e4066cd14aa18619310a4490246b P 8398e6f89bf5419d81ed5bed991da0cf -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:33937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:28.670300 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 13126
W20240503 08:05:28.676813 14722 connection.cc:535] client connection to 127.0.99.132:37969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:05:28.677412 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 14576
W20240503 08:05:28.678165 14722 consensus_peers.cc:485] T 5212dff7ded143a7923bb6176466b0fa P b052e89b63c24258872c832a6a0d4d4d -> Peer af712db2f85249adbe718e8ba0549f84 (127.0.99.132:37969): Couldn't send request to peer af712db2f85249adbe718e8ba0549f84. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.132:37969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:28.681600 14725 consensus_peers.cc:485] T 9c7bf40aa0614923a19feed8bab430fa P b052e89b63c24258872c832a6a0d4d4d -> Peer 0cf63591cfc244028e1fc368c2bca079 (127.0.99.130:33937): Couldn't send request to peer 0cf63591cfc244028e1fc368c2bca079. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:33937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:28.684089 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 14707
I20240503 08:05:28.690163 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 12642
2024-05-03T08:05:28Z chronyd exiting
I20240503 08:05:28.706892 398 test_util.cc:175] -----------------------------------------------
I20240503 08:05:28.706969 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714723107332537-398-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (329389 ms)
[----------] 1 test from ConcurrentRebalancersTest (329390 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240503 08:05:28.707571 398 test_util.cc:268] Using random seed: 785452529
I20240503 08:05:28.708240 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:05:28.708276 398 ts_itest-base.cc:113] --------------
I20240503 08:05:28.708284 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:05:28.708293 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:05:28.708302 398 ts_itest-base.cc:116] --------------
2024-05-03T08:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:05:28Z Disabled control of system clock
I20240503 08:05:28.715538 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38799
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38799
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 08:05:28.820814 18137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:28.821034 18137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:28.821089 18137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:28.824093 18137 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:28.824162 18137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:05:28.824193 18137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:28.824225 18137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:05:28.824257 18137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:05:28.827868 18137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38799
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:28.828750 18137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:28.829838 18137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:05:28.837643 18137 server_base.cc:1034] running on GCE node
W20240503 08:05:28.838188 18142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:28.838593 18145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:28.840216 18143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:28.840727 18137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:28.841606 18137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:28.842767 18137 hybrid_clock.cc:648] HybridClock initialized: now 1714723528842740 us; error 36 us; skew 500 ppm
I20240503 08:05:28.844810 18137 webserver.cc:460] Webserver started at http://127.0.99.190:40231/ using document root <none> and password file <none>
I20240503 08:05:28.845371 18137 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:28.845430 18137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:28.845654 18137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:28.847402 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a"
format_stamp: "Formatted at 2024-05-03 08:05:28 on dist-test-slave-11df"
I20240503 08:05:28.847878 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a"
format_stamp: "Formatted at 2024-05-03 08:05:28 on dist-test-slave-11df"
I20240503 08:05:28.851162 18137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:05:28.853353 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:28.854329 18137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:28.854445 18137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a"
format_stamp: "Formatted at 2024-05-03 08:05:28 on dist-test-slave-11df"
I20240503 08:05:28.854523 18137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:28.890558 18137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:28.891305 18137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:28.891474 18137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:28.897776 18137 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38799
I20240503 08:05:28.897797 18203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38799 every 8 connection(s)
I20240503 08:05:28.898712 18137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:05:28.900418 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18137
I20240503 08:05:28.900542 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:05:28.901336 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:28.907411 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a: Bootstrap starting.
I20240503 08:05:28.909648 18204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:28.910426 18204 log.cc:824] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:28.912016 18204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a: No bootstrap required, opened a new log
I20240503 08:05:28.914573 18204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } }
I20240503 08:05:28.914752 18204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:28.914788 18204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c67ddb4fa7f48e2aa750d2ad0d60b3a, State: Initialized, Role: FOLLOWER
I20240503 08:05:28.915169 18204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } }
I20240503 08:05:28.915279 18204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:28.915316 18204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:28.915398 18204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:28.916076 18204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } }
I20240503 08:05:28.916353 18204 leader_election.cc:302] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c67ddb4fa7f48e2aa750d2ad0d60b3a; no voters:
I20240503 08:05:28.916594 18204 leader_election.cc:288] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:28.916939 18209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:28.917119 18209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [term 1 LEADER]: Becoming Leader. State: Replica: 9c67ddb4fa7f48e2aa750d2ad0d60b3a, State: Running, Role: LEADER
I20240503 08:05:28.917393 18209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } }
I20240503 08:05:28.917618 18204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:05:28.918820 18211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c67ddb4fa7f48e2aa750d2ad0d60b3a. Latest consensus state: current_term: 1 leader_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } } }
I20240503 08:05:28.918964 18211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:28.918813 18210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c67ddb4fa7f48e2aa750d2ad0d60b3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38799 } } }
I20240503 08:05:28.919174 18210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:28.919523 18219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:05:28.922034 18219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:05:28.927312 18219 catalog_manager.cc:1347] Generated new cluster ID: 4aa4d295e8c746eb979d84d4ace8585e
I20240503 08:05:28.927376 18219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:05:28.946039 18219 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:05:28.947227 18219 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:05:28.963559 18219 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9c67ddb4fa7f48e2aa750d2ad0d60b3a: Generated new TSK 0
I20240503 08:05:28.964812 18219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:05:28.979462 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:05:29.108286 18228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:29.108497 18228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:29.108553 18228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:29.111537 18228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:29.111608 18228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:29.111689 18228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:05:29.115226 18228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.18228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:29.116029 18228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:29.117086 18228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:29.125335 18233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.126080 18234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.126703 18236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:29.127298 18228 server_base.cc:1034] running on GCE node
I20240503 08:05:29.127645 18228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:29.128144 18228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:29.129279 18228 hybrid_clock.cc:648] HybridClock initialized: now 1714723529129268 us; error 31 us; skew 500 ppm
I20240503 08:05:29.131124 18228 webserver.cc:460] Webserver started at http://127.0.99.129:35391/ using document root <none> and password file <none>
I20240503 08:05:29.131646 18228 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:29.131703 18228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:29.131860 18228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:29.136020 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "89e247cded31418995fcbce06a3b8eb0"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.136464 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "89e247cded31418995fcbce06a3b8eb0"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.139519 18228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:05:29.141507 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.142436 18228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:29.142532 18228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "89e247cded31418995fcbce06a3b8eb0"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.142606 18228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:29.154189 18228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:29.154728 18228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:29.154875 18228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:29.155416 18228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:29.156251 18228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:29.156298 18228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.156354 18228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:29.156378 18228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.164490 18228 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46747
I20240503 08:05:29.164523 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46747 every 8 connection(s)
I20240503 08:05:29.165490 18228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:05:29.173856 18356 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:29.174069 18356 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:29.174492 18356 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:29.175642 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18228
I20240503 08:05:29.175748 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:05:29.175801 18168 ts_manager.cc:194] Registered new tserver with Master: 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129:46747)
I20240503 08:05:29.176987 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:58563
I20240503 08:05:29.177721 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:05:29.282298 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:29.282547 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:29.282616 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:29.285694 18359 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:29.285780 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:29.285878 18359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:05:29.289527 18359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.18359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:29.290444 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:29.291623 18359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:29.300065 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.300377 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.301026 18364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:29.301736 18359 server_base.cc:1034] running on GCE node
I20240503 08:05:29.302145 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:29.302726 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:29.303891 18359 hybrid_clock.cc:648] HybridClock initialized: now 1714723529303879 us; error 29 us; skew 500 ppm
I20240503 08:05:29.305779 18359 webserver.cc:460] Webserver started at http://127.0.99.130:40113/ using document root <none> and password file <none>
I20240503 08:05:29.306304 18359 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:29.306365 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:29.306528 18359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:29.308161 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "36ca07763dce4f20ad27261249a3e74f"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.308614 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "36ca07763dce4f20ad27261249a3e74f"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.311805 18359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:05:29.313769 18373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.314755 18359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:29.314858 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "36ca07763dce4f20ad27261249a3e74f"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.314970 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:29.341006 18359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:29.341593 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:29.341737 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:29.342244 18359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:29.343092 18359 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:29.343137 18359 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.343189 18359 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:29.343214 18359 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.351141 18359 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:45125
I20240503 08:05:29.351171 18486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:45125 every 8 connection(s)
I20240503 08:05:29.352066 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:05:29.352855 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18359
I20240503 08:05:29.352984 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:05:29.354964 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:05:29.361019 18487 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:29.361220 18487 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:29.361639 18487 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:29.362530 18168 ts_manager.cc:194] Registered new tserver with Master: 36ca07763dce4f20ad27261249a3e74f (127.0.99.130:45125)
I20240503 08:05:29.363085 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52385
W20240503 08:05:29.461876 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:29.462113 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:29.462180 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:29.465174 18490 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:29.465243 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:29.465332 18490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:05:29.468942 18490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.18490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:29.469798 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:29.470886 18490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:29.478756 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.479379 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.479960 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:29.480574 18490 server_base.cc:1034] running on GCE node
I20240503 08:05:29.480932 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:29.481446 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:29.482589 18490 hybrid_clock.cc:648] HybridClock initialized: now 1714723529482570 us; error 33 us; skew 500 ppm
I20240503 08:05:29.484386 18490 webserver.cc:460] Webserver started at http://127.0.99.131:32929/ using document root <none> and password file <none>
I20240503 08:05:29.484891 18490 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:29.484943 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:29.485101 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:29.486682 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2e7dbe6b7496469e8ecfe68d24440a2e"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.487188 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2e7dbe6b7496469e8ecfe68d24440a2e"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.490391 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:05:29.492475 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.493448 18490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:29.493546 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2e7dbe6b7496469e8ecfe68d24440a2e"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.493628 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:29.504490 18490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:29.504982 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:29.505115 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:29.505565 18490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:29.506384 18490 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:29.506428 18490 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.506475 18490 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:29.506500 18490 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.514101 18490 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:32897
I20240503 08:05:29.514146 18617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:32897 every 8 connection(s)
I20240503 08:05:29.514981 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:05:29.520749 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18490
I20240503 08:05:29.520861 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:05:29.523072 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:05:29.523216 18618 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:29.523440 18618 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:29.523844 18618 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:29.524832 18168 ts_manager.cc:194] Registered new tserver with Master: 2e7dbe6b7496469e8ecfe68d24440a2e (127.0.99.131:32897)
I20240503 08:05:29.525333 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:53873
W20240503 08:05:29.626765 18621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:29.626991 18621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:29.627049 18621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:29.629925 18621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:29.629981 18621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:29.630043 18621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:29.633450 18621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.18621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:29.634219 18621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:29.635303 18621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:29.643159 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.643817 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:29.644454 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:29.645112 18621 server_base.cc:1034] running on GCE node
I20240503 08:05:29.645479 18621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:29.645977 18621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:29.647119 18621 hybrid_clock.cc:648] HybridClock initialized: now 1714723529647102 us; error 36 us; skew 500 ppm
I20240503 08:05:29.648910 18621 webserver.cc:460] Webserver started at http://127.0.99.132:35849/ using document root <none> and password file <none>
I20240503 08:05:29.649449 18621 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:29.649505 18621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:29.649667 18621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:29.651302 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0de18b3d5b254048946b0f3bd4d46dff"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.651721 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0de18b3d5b254048946b0f3bd4d46dff"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.654697 18621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:05:29.656685 18635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.657606 18621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:29.657702 18621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0de18b3d5b254048946b0f3bd4d46dff"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:29.657776 18621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:29.678126 18621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:29.678687 18621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:29.678831 18621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:29.679345 18621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:29.680179 18621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:29.680223 18621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.680275 18621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:29.680300 18621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:29.687834 18621 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45701
I20240503 08:05:29.687852 18748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45701 every 8 connection(s)
I20240503 08:05:29.688679 18621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:29.689112 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18621
I20240503 08:05:29.689232 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:05:29.696738 18749 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:29.696938 18749 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:29.697511 18749 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:29.698348 18168 ts_manager.cc:194] Registered new tserver with Master: 0de18b3d5b254048946b0f3bd4d46dff (127.0.99.132:45701)
I20240503 08:05:29.698854 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45847
I20240503 08:05:29.703073 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:05:29.711097 18168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43366:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:05:29.726155 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 93c0d5b9d25145ae826b6fcece0ab75a (DEFAULT_TABLE table=TestTable [id=1e41b8bdc34440f6be2f0c531ce3bf50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:29.727025 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c0d5b9d25145ae826b6fcece0ab75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:29.731891 18762 tablet_bootstrap.cc:492] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:29.733711 18762 tablet_bootstrap.cc:654] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:29.734336 18762 log.cc:824] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:29.735920 18762 tablet_bootstrap.cc:492] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:29.736119 18762 ts_tablet_manager.cc:1367] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:05:29.739045 18762 raft_consensus.cc:348] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:29.739225 18762 raft_consensus.cc:374] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:29.739279 18762 raft_consensus.cc:729] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:29.739694 18762 consensus_queue.cc:260] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:29.739821 18762 raft_consensus.cc:388] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:29.739886 18762 raft_consensus.cc:482] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:29.739990 18762 raft_consensus.cc:3037] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:29.740799 18762 raft_consensus.cc:504] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:29.741117 18762 leader_election.cc:302] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:29.741405 18762 leader_election.cc:288] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:29.741500 18764 raft_consensus.cc:2781] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:29.741669 18764 raft_consensus.cc:686] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:29.741940 18764 consensus_queue.cc:237] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:29.742075 18356 heartbeater.cc:503] Master 127.0.99.190:38799 was elected leader, sending a full tablet report...
I20240503 08:05:29.742300 18762 ts_tablet_manager.cc:1398] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 08:05:29.745143 18168 catalog_manager.cc:5484] T 93c0d5b9d25145ae826b6fcece0ab75a P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:29.752447 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:05:29.753700 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 89e247cded31418995fcbce06a3b8eb0 to finish bootstrapping
I20240503 08:05:29.758618 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 36ca07763dce4f20ad27261249a3e74f to finish bootstrapping
I20240503 08:05:29.764086 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2e7dbe6b7496469e8ecfe68d24440a2e to finish bootstrapping
I20240503 08:05:29.769407 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0de18b3d5b254048946b0f3bd4d46dff to finish bootstrapping
I20240503 08:05:29.774515 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18490
I20240503 08:05:29.779681 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18621
I20240503 08:05:30.364817 18487 heartbeater.cc:503] Master 127.0.99.190:38799 was elected leader, sending a full tablet report...
I20240503 08:05:33.535382 18168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43366:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:05:33.542603 18290 tablet_service.cc:1467] Processing CreateTablet for tablet fe8743c8ac4f45bc82278a08bfa56ebf (DEFAULT_TABLE table=rebalance_test_table_0 [id=6a08e1c37d474b7387222fac18ebd542]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.542945 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8743c8ac4f45bc82278a08bfa56ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.545068 18779 tablet_bootstrap.cc:492] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:33.545939 18779 tablet_bootstrap.cc:654] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.547444 18779 tablet_bootstrap.cc:492] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:33.547533 18779 ts_tablet_manager.cc:1367] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:33.547451 18421 tablet_service.cc:1467] Processing CreateTablet for tablet c810c5587b3045588ce3bc0530ba628d (DEFAULT_TABLE table=rebalance_test_table_0 [id=6a08e1c37d474b7387222fac18ebd542]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.547439 18420 tablet_service.cc:1467] Processing CreateTablet for tablet 88f383179aaf45f993b9a2daa7713928 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6a08e1c37d474b7387222fac18ebd542]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.548043 18779 raft_consensus.cc:348] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.548130 18779 raft_consensus.cc:374] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.548159 18779 raft_consensus.cc:729] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.548280 18779 consensus_queue.cc:260] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.548359 18779 raft_consensus.cc:388] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.548378 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f383179aaf45f993b9a2daa7713928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.548396 18779 raft_consensus.cc:482] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.548436 18779 raft_consensus.cc:3037] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.548754 18421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c810c5587b3045588ce3bc0530ba628d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.549083 18779 raft_consensus.cc:504] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.549203 18779 leader_election.cc:302] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:33.549346 18779 leader_election.cc:288] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.549477 18781 raft_consensus.cc:2781] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.549614 18779 ts_tablet_manager.cc:1398] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:05:33.549726 18781 raft_consensus.cc:686] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:33.549870 18781 consensus_queue.cc:237] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.551090 18168 catalog_manager.cc:5484] T fe8743c8ac4f45bc82278a08bfa56ebf P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:33.554118 18786 tablet_bootstrap.cc:492] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:33.555848 18786 tablet_bootstrap.cc:654] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.556443 18786 log.cc:824] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:33.557754 18786 tablet_bootstrap.cc:492] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:33.557885 18786 ts_tablet_manager.cc:1367] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:05:33.560622 18786 raft_consensus.cc:348] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.560777 18786 raft_consensus.cc:374] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.560815 18786 raft_consensus.cc:729] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.561156 18786 consensus_queue.cc:260] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.561262 18786 raft_consensus.cc:388] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.561305 18786 raft_consensus.cc:482] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.561386 18786 raft_consensus.cc:3037] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.562041 18786 raft_consensus.cc:504] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.562314 18786 leader_election.cc:302] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:33.562539 18786 leader_election.cc:288] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.562654 18788 raft_consensus.cc:2781] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.562870 18788 raft_consensus.cc:686] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:33.563191 18788 consensus_queue.cc:237] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.563279 18786 ts_tablet_manager.cc:1398] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 08:05:33.563480 18786 tablet_bootstrap.cc:492] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:33.564235 18786 tablet_bootstrap.cc:654] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.565737 18168 catalog_manager.cc:5484] T c810c5587b3045588ce3bc0530ba628d P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:33.566074 18786 tablet_bootstrap.cc:492] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:33.566146 18786 ts_tablet_manager.cc:1367] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:33.566569 18786 raft_consensus.cc:348] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.566669 18786 raft_consensus.cc:374] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.566700 18786 raft_consensus.cc:729] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.566810 18786 consensus_queue.cc:260] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.566885 18786 raft_consensus.cc:388] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.566943 18786 raft_consensus.cc:482] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.566990 18786 raft_consensus.cc:3037] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.567704 18786 raft_consensus.cc:504] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.567812 18786 leader_election.cc:302] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:33.567896 18786 leader_election.cc:288] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.567930 18788 raft_consensus.cc:2781] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.568007 18788 raft_consensus.cc:686] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:33.568107 18786 ts_tablet_manager.cc:1398] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:05:33.568118 18788 consensus_queue.cc:237] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.569754 18168 catalog_manager.cc:5484] T 88f383179aaf45f993b9a2daa7713928 P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:33.729257 18167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43366:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:05:33.734243 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 3496f7c076934214a2f1a855fe77cd44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3fa2daf29b3d4d8da4feecd5089d0dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.734283 18421 tablet_service.cc:1467] Processing CreateTablet for tablet 11ed0dbe7a5246cf80bff568d93aea59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3fa2daf29b3d4d8da4feecd5089d0dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.734252 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 6d61e0d619434162a70b8e7418f39313 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3fa2daf29b3d4d8da4feecd5089d0dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.734567 18421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ed0dbe7a5246cf80bff568d93aea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.734571 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3496f7c076934214a2f1a855fe77cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.734776 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d61e0d619434162a70b8e7418f39313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.736523 18786 tablet_bootstrap.cc:492] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:33.736799 18779 tablet_bootstrap.cc:492] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:33.737450 18786 tablet_bootstrap.cc:654] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.737582 18779 tablet_bootstrap.cc:654] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.738603 18779 tablet_bootstrap.cc:492] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:33.738674 18779 ts_tablet_manager.cc:1367] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:05:33.738811 18786 tablet_bootstrap.cc:492] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:33.738879 18786 ts_tablet_manager.cc:1367] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:05:33.739115 18779 raft_consensus.cc:348] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.739190 18779 raft_consensus.cc:374] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.739210 18779 raft_consensus.cc:729] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.739305 18779 consensus_queue.cc:260] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.739370 18779 raft_consensus.cc:388] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.739392 18779 raft_consensus.cc:482] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.739418 18779 raft_consensus.cc:3037] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.739450 18786 raft_consensus.cc:348] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.739533 18786 raft_consensus.cc:374] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.739557 18786 raft_consensus.cc:729] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.739672 18786 consensus_queue.cc:260] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.739745 18786 raft_consensus.cc:388] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.739771 18786 raft_consensus.cc:482] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.739802 18786 raft_consensus.cc:3037] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.739979 18779 raft_consensus.cc:504] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.740074 18779 leader_election.cc:302] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:33.740142 18779 leader_election.cc:288] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.740242 18783 raft_consensus.cc:2781] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.740348 18783 raft_consensus.cc:686] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:33.740470 18783 consensus_queue.cc:237] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.740545 18786 raft_consensus.cc:504] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.740646 18786 leader_election.cc:302] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:33.740717 18786 leader_election.cc:288] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.740779 18788 raft_consensus.cc:2781] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.740868 18788 raft_consensus.cc:686] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:33.740980 18788 consensus_queue.cc:237] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.741091 18786 ts_tablet_manager.cc:1398] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.740345 18779 ts_tablet_manager.cc:1398] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:05:33.741689 18167 catalog_manager.cc:5484] T 3496f7c076934214a2f1a855fe77cd44 P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:33.741814 18779 tablet_bootstrap.cc:492] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:33.742305 18168 catalog_manager.cc:5484] T 11ed0dbe7a5246cf80bff568d93aea59 P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:33.744414 18779 tablet_bootstrap.cc:654] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.745365 18779 tablet_bootstrap.cc:492] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:33.745438 18779 ts_tablet_manager.cc:1367] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:05:33.745764 18779 raft_consensus.cc:348] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.745837 18779 raft_consensus.cc:374] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.745857 18779 raft_consensus.cc:729] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.745945 18779 consensus_queue.cc:260] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.746008 18779 raft_consensus.cc:388] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.746035 18779 raft_consensus.cc:482] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.746066 18779 raft_consensus.cc:3037] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.746769 18779 raft_consensus.cc:504] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.746891 18779 leader_election.cc:302] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:33.747017 18779 leader_election.cc:288] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.747120 18781 raft_consensus.cc:2781] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.747195 18781 raft_consensus.cc:686] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:33.747221 18779 ts_tablet_manager.cc:1398] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.747298 18781 consensus_queue.cc:237] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.748347 18168 catalog_manager.cc:5484] T 6d61e0d619434162a70b8e7418f39313 P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:33.905218 18168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43366:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:05:33.910451 18420 tablet_service.cc:1467] Processing CreateTablet for tablet d47332a5fbb84f3c81819aec95b84dd6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c68058163e74ccc956c77d7c314c144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.910439 18421 tablet_service.cc:1467] Processing CreateTablet for tablet 9934be96ca4a4f14a65bc1461ad76586 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c68058163e74ccc956c77d7c314c144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.910513 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 379943c4d23a42e8960ea3e5d9ffe37d (DEFAULT_TABLE table=rebalance_test_table_2 [id=8c68058163e74ccc956c77d7c314c144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:33.910790 18421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9934be96ca4a4f14a65bc1461ad76586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.910791 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379943c4d23a42e8960ea3e5d9ffe37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.911010 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47332a5fbb84f3c81819aec95b84dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:33.912431 18779 tablet_bootstrap.cc:492] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:33.913053 18786 tablet_bootstrap.cc:492] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:33.913350 18779 tablet_bootstrap.cc:654] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.913883 18786 tablet_bootstrap.cc:654] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.914422 18779 tablet_bootstrap.cc:492] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:33.914510 18779 ts_tablet_manager.cc:1367] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:33.914875 18786 tablet_bootstrap.cc:492] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:33.914870 18779 raft_consensus.cc:348] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.914976 18779 raft_consensus.cc:374] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.914986 18786 ts_tablet_manager.cc:1367] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.915019 18779 raft_consensus.cc:729] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.915148 18779 consensus_queue.cc:260] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.915227 18779 raft_consensus.cc:388] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.915261 18779 raft_consensus.cc:482] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.915303 18779 raft_consensus.cc:3037] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.915409 18786 raft_consensus.cc:348] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.915498 18786 raft_consensus.cc:374] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.915534 18786 raft_consensus.cc:729] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.915643 18786 consensus_queue.cc:260] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.915724 18786 raft_consensus.cc:388] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.915752 18786 raft_consensus.cc:482] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.915781 18786 raft_consensus.cc:3037] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.916115 18779 raft_consensus.cc:504] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.916236 18779 leader_election.cc:302] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:33.916328 18779 leader_election.cc:288] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.916397 18781 raft_consensus.cc:2781] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.916461 18786 raft_consensus.cc:504] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.916488 18781 raft_consensus.cc:686] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:33.916564 18786 leader_election.cc:302] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:33.916576 18779 ts_tablet_manager.cc:1398] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.916642 18786 leader_election.cc:288] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.916587 18781 consensus_queue.cc:237] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:33.916772 18790 raft_consensus.cc:2781] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.916851 18790 raft_consensus.cc:686] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:33.916998 18786 ts_tablet_manager.cc:1398] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.916985 18790 consensus_queue.cc:237] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.917183 18786 tablet_bootstrap.cc:492] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:33.917918 18168 catalog_manager.cc:5484] T 379943c4d23a42e8960ea3e5d9ffe37d P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:33.918215 18167 catalog_manager.cc:5484] T 9934be96ca4a4f14a65bc1461ad76586 P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:33.918272 18786 tablet_bootstrap.cc:654] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:33.920485 18786 tablet_bootstrap.cc:492] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:33.920552 18786 ts_tablet_manager.cc:1367] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:33.920909 18786 raft_consensus.cc:348] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.921010 18786 raft_consensus.cc:374] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:33.921051 18786 raft_consensus.cc:729] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:33.921151 18786 consensus_queue.cc:260] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.921218 18786 raft_consensus.cc:388] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:33.921252 18786 raft_consensus.cc:482] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:33.921288 18786 raft_consensus.cc:3037] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:33.921990 18786 raft_consensus.cc:504] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.922109 18786 leader_election.cc:302] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:33.922179 18786 leader_election.cc:288] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:33.922232 18790 raft_consensus.cc:2781] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:33.922305 18790 raft_consensus.cc:686] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:33.922400 18786 ts_tablet_manager.cc:1398] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:33.922403 18790 consensus_queue.cc:237] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:33.923527 18168 catalog_manager.cc:5484] T d47332a5fbb84f3c81819aec95b84dd6 P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:34.076727 18168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43366:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:05:34.081743 18421 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd9357f60ab4195aafc5a33c00b5193 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed2b505086404202ba25b4327fbc5494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:34.081722 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 361bb754d5f94e908bf8ac97a5a49cd5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed2b505086404202ba25b4327fbc5494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:34.082041 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361bb754d5f94e908bf8ac97a5a49cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:34.082051 18421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd9357f60ab4195aafc5a33c00b5193. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:34.083725 18786 tablet_bootstrap.cc:492] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f: Bootstrap starting.
I20240503 08:05:34.083726 18779 tablet_bootstrap.cc:492] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:34.081722 18290 tablet_service.cc:1467] Processing CreateTablet for tablet ef06e09c2b504440a88eba7e268c81d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed2b505086404202ba25b4327fbc5494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:34.084008 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06e09c2b504440a88eba7e268c81d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:34.084609 18779 tablet_bootstrap.cc:654] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:34.084609 18786 tablet_bootstrap.cc:654] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:34.085856 18786 tablet_bootstrap.cc:492] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f: No bootstrap required, opened a new log
I20240503 08:05:34.085925 18786 ts_tablet_manager.cc:1367] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:34.086282 18786 raft_consensus.cc:348] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:34.086369 18786 raft_consensus.cc:374] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:34.086398 18786 raft_consensus.cc:729] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Initialized, Role: FOLLOWER
I20240503 08:05:34.086490 18786 consensus_queue.cc:260] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:34.086544 18786 raft_consensus.cc:388] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:34.086570 18786 raft_consensus.cc:482] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:34.086602 18786 raft_consensus.cc:3037] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:34.087239 18779 tablet_bootstrap.cc:492] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:34.087266 18786 raft_consensus.cc:504] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:34.087322 18779 ts_tablet_manager.cc:1367] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:05:34.087389 18786 leader_election.cc:302] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ca07763dce4f20ad27261249a3e74f; no voters:
I20240503 08:05:34.087491 18786 leader_election.cc:288] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:34.087539 18790 raft_consensus.cc:2781] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:34.087635 18790 raft_consensus.cc:686] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [term 1 LEADER]: Becoming Leader. State: Replica: 36ca07763dce4f20ad27261249a3e74f, State: Running, Role: LEADER
I20240503 08:05:34.087766 18786 ts_tablet_manager.cc:1398] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:34.087736 18779 raft_consensus.cc:348] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.087741 18790 consensus_queue.cc:237] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } }
I20240503 08:05:34.087816 18779 raft_consensus.cc:374] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:34.087852 18779 raft_consensus.cc:729] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:34.087961 18779 consensus_queue.cc:260] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.088037 18779 raft_consensus.cc:388] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:34.088073 18779 raft_consensus.cc:482] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:34.088109 18779 raft_consensus.cc:3037] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:34.088726 18779 raft_consensus.cc:504] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.088833 18779 leader_election.cc:302] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:34.088915 18779 leader_election.cc:288] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:34.088973 18781 raft_consensus.cc:2781] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:34.089097 18779 ts_tablet_manager.cc:1398] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:34.089107 18781 raft_consensus.cc:686] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:34.089231 18779 tablet_bootstrap.cc:492] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0: Bootstrap starting.
I20240503 08:05:34.089224 18781 consensus_queue.cc:237] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.090242 18168 catalog_manager.cc:5484] T 2cd9357f60ab4195aafc5a33c00b5193 P 36ca07763dce4f20ad27261249a3e74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ca07763dce4f20ad27261249a3e74f (127.0.99.130). New cstate: current_term: 1 leader_uuid: "36ca07763dce4f20ad27261249a3e74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ca07763dce4f20ad27261249a3e74f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45125 } } }
I20240503 08:05:34.090427 18779 tablet_bootstrap.cc:654] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:34.091229 18167 catalog_manager.cc:5484] T 361bb754d5f94e908bf8ac97a5a49cd5 P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:34.091609 18779 tablet_bootstrap.cc:492] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0: No bootstrap required, opened a new log
I20240503 08:05:34.091679 18779 ts_tablet_manager.cc:1367] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:34.092043 18779 raft_consensus.cc:348] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.092128 18779 raft_consensus.cc:374] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:34.092156 18779 raft_consensus.cc:729] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Initialized, Role: FOLLOWER
I20240503 08:05:34.092260 18779 consensus_queue.cc:260] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.092325 18779 raft_consensus.cc:388] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:34.092358 18779 raft_consensus.cc:482] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:34.092401 18779 raft_consensus.cc:3037] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:34.093161 18779 raft_consensus.cc:504] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.093271 18779 leader_election.cc:302] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e247cded31418995fcbce06a3b8eb0; no voters:
I20240503 08:05:34.093354 18779 leader_election.cc:288] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:34.093505 18782 raft_consensus.cc:2781] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:34.093546 18779 ts_tablet_manager.cc:1398] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:05:34.093598 18782 raft_consensus.cc:686] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 89e247cded31418995fcbce06a3b8eb0, State: Running, Role: LEADER
I20240503 08:05:34.093724 18782 consensus_queue.cc:237] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } }
I20240503 08:05:34.095062 18167 catalog_manager.cc:5484] T ef06e09c2b504440a88eba7e268c81d1 P 89e247cded31418995fcbce06a3b8eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e247cded31418995fcbce06a3b8eb0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "89e247cded31418995fcbce06a3b8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e247cded31418995fcbce06a3b8eb0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46747 } } }
I20240503 08:05:34.245231 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:32897
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:38799
--webserver_port=32929
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 08:05:34.346845 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:34.347105 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:34.347157 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:34.350137 18886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:34.350198 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:34.350283 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:05:34.353900 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:32897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=32929
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:34.354785 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:34.355916 18886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:34.364267 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:34.364601 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:34.365234 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:34.365872 18886 server_base.cc:1034] running on GCE node
I20240503 08:05:34.366262 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:34.366801 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:34.367945 18886 hybrid_clock.cc:648] HybridClock initialized: now 1714723534367920 us; error 47 us; skew 500 ppm
I20240503 08:05:34.369803 18886 webserver.cc:460] Webserver started at http://127.0.99.131:32929/ using document root <none> and password file <none>
I20240503 08:05:34.370342 18886 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:34.370414 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:34.373597 18886 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:34.375380 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.376323 18886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:34.376420 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2e7dbe6b7496469e8ecfe68d24440a2e"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:34.376827 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:34.418236 18886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:34.418911 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:34.419111 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:34.419636 18886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:34.420513 18886 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:34.420559 18886 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.420612 18886 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:34.420637 18886 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.428479 18886 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:32897
I20240503 08:05:34.428519 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:32897 every 8 connection(s)
I20240503 08:05:34.429487 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:05:34.430372 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 18886
I20240503 08:05:34.430765 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:45701
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:38799
--webserver_port=35849
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 08:05:34.438534 19014 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:34.438743 19014 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:34.439237 19014 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:34.440343 18167 ts_manager.cc:194] Re-registered known tserver with Master: 2e7dbe6b7496469e8ecfe68d24440a2e (127.0.99.131:32897)
I20240503 08:05:34.441089 18167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49351
W20240503 08:05:34.533633 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:34.533828 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:34.533867 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:34.536763 19017 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 08:05:34.536820 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:34.536882 19017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:34.540309 19017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:45701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=35849
--tserver_master_addrs=127.0.99.190:38799
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:34.541087 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:34.542091 19017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:34.549803 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:34.550514 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:34.551100 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:34.551708 19017 server_base.cc:1034] running on GCE node
I20240503 08:05:34.552033 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:34.552557 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:34.553750 19017 hybrid_clock.cc:648] HybridClock initialized: now 1714723534553726 us; error 27 us; skew 500 ppm
I20240503 08:05:34.555712 19017 webserver.cc:460] Webserver started at http://127.0.99.132:35849/ using document root <none> and password file <none>
I20240503 08:05:34.556300 19017 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:34.556432 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:34.559739 19017 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:05:34.561564 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.562565 19017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:05:34.562661 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0de18b3d5b254048946b0f3bd4d46dff"
format_stamp: "Formatted at 2024-05-03 08:05:29 on dist-test-slave-11df"
I20240503 08:05:34.563071 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:34.607218 19017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:34.607944 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:34.608135 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:34.608767 19017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:34.609691 19017 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:34.609741 19017 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.609792 19017 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:34.609817 19017 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:34.618499 19017 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45701
I20240503 08:05:34.618531 19144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45701 every 8 connection(s)
I20240503 08:05:34.619712 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:34.626533 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19017
I20240503 08:05:34.628110 19145 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38799
I20240503 08:05:34.628376 19145 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:34.628893 19145 heartbeater.cc:511] Master 127.0.99.190:38799 requested a full tablet report, sending...
I20240503 08:05:34.629843 18167 ts_manager.cc:194] Re-registered known tserver with Master: 0de18b3d5b254048946b0f3bd4d46dff (127.0.99.132:45701)
I20240503 08:05:34.630610 18167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55811
I20240503 08:05:34.808197 18290 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:34.808943 19079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:34.809128 18421 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:34.809970 18948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:34.906046 18948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:34.906556 18421 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:34.909368 18290 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:34.910054 19079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:35.008282 19079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:35.008554 18948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:35.011087 18421 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:35.011791 18290 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:35.097887 18948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:35.098253 19079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:35.099102 18290 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:05:35.119359 18421 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
9c67ddb4fa7f48e2aa750d2ad0d60b3a | 127.0.99.190:38799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42213 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0de18b3d5b254048946b0f3bd4d46dff | 127.0.99.132:45701 | HEALTHY | <none> | 0 | 0
2e7dbe6b7496469e8ecfe68d24440a2e | 127.0.99.131:32897 | HEALTHY | <none> | 0 | 0
36ca07763dce4f20ad27261249a3e74f | 127.0.99.130:45125 | HEALTHY | <none> | 6 | 0
89e247cded31418995fcbce06a3b8eb0 | 127.0.99.129:46747 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:46747
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:45125
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:32897
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:45701
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:46747
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:45125
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:32897
server_dump_info_path | /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:45701
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42213 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:05:35.137946 398 log_verifier.cc:126] Checking tablet 11ed0dbe7a5246cf80bff568d93aea59
I20240503 08:05:35.145298 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 11ed0dbe7a5246cf80bff568d93aea59
I20240503 08:05:35.145352 398 log_verifier.cc:126] Checking tablet 2cd9357f60ab4195aafc5a33c00b5193
I20240503 08:05:35.152153 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2cd9357f60ab4195aafc5a33c00b5193
I20240503 08:05:35.152205 398 log_verifier.cc:126] Checking tablet 3496f7c076934214a2f1a855fe77cd44
I20240503 08:05:35.159063 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3496f7c076934214a2f1a855fe77cd44
I20240503 08:05:35.159112 398 log_verifier.cc:126] Checking tablet 361bb754d5f94e908bf8ac97a5a49cd5
I20240503 08:05:35.166106 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 361bb754d5f94e908bf8ac97a5a49cd5
I20240503 08:05:35.166162 398 log_verifier.cc:126] Checking tablet 379943c4d23a42e8960ea3e5d9ffe37d
I20240503 08:05:35.174221 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 379943c4d23a42e8960ea3e5d9ffe37d
I20240503 08:05:35.174276 398 log_verifier.cc:126] Checking tablet 6d61e0d619434162a70b8e7418f39313
I20240503 08:05:35.181869 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6d61e0d619434162a70b8e7418f39313
I20240503 08:05:35.181922 398 log_verifier.cc:126] Checking tablet 88f383179aaf45f993b9a2daa7713928
I20240503 08:05:35.188728 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 88f383179aaf45f993b9a2daa7713928
I20240503 08:05:35.188784 398 log_verifier.cc:126] Checking tablet 93c0d5b9d25145ae826b6fcece0ab75a
I20240503 08:05:35.195523 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 93c0d5b9d25145ae826b6fcece0ab75a
I20240503 08:05:35.195580 398 log_verifier.cc:126] Checking tablet 9934be96ca4a4f14a65bc1461ad76586
I20240503 08:05:35.202567 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9934be96ca4a4f14a65bc1461ad76586
I20240503 08:05:35.202628 398 log_verifier.cc:126] Checking tablet c810c5587b3045588ce3bc0530ba628d
I20240503 08:05:35.209628 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c810c5587b3045588ce3bc0530ba628d
I20240503 08:05:35.209684 398 log_verifier.cc:126] Checking tablet d47332a5fbb84f3c81819aec95b84dd6
I20240503 08:05:35.216377 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d47332a5fbb84f3c81819aec95b84dd6
I20240503 08:05:35.216434 398 log_verifier.cc:126] Checking tablet ef06e09c2b504440a88eba7e268c81d1
I20240503 08:05:35.223305 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ef06e09c2b504440a88eba7e268c81d1
I20240503 08:05:35.223363 398 log_verifier.cc:126] Checking tablet fe8743c8ac4f45bc82278a08bfa56ebf
I20240503 08:05:35.229988 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fe8743c8ac4f45bc82278a08bfa56ebf
I20240503 08:05:35.230438 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18228
I20240503 08:05:35.235854 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18359
I20240503 08:05:35.240952 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18886
I20240503 08:05:35.245707 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19017
I20240503 08:05:35.252007 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 18137
2024-05-03T08:05:35Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (6576 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6576 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240503 08:05:35.284581 398 test_util.cc:268] Using random seed: 792029536
I20240503 08:05:35.285326 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:05:35.285363 398 ts_itest-base.cc:113] --------------
I20240503 08:05:35.285374 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:05:35.285383 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:05:35.285392 398 ts_itest-base.cc:116] --------------
2024-05-03T08:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:05:35Z Disabled control of system clock
I20240503 08:05:35.293274 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43309
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43309
--location_mapping_cmd=/tmp/dist-test-task8ZGZ68/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:05:35.396263 19291 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:05:35.396520 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:35.396562 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:35.396620 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:35.399667 19291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:35.399734 19291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:05:35.399758 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:35.399780 19291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:05:35.399811 19291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:05:35.403430 19291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task8ZGZ68/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43309
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:35.404292 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:35.405413 19291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:35.412865 19296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.413085 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.413735 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:35.414391 19291 server_base.cc:1034] running on GCE node
I20240503 08:05:35.414958 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:35.415932 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:35.417085 19291 hybrid_clock.cc:648] HybridClock initialized: now 1714723535417065 us; error 36 us; skew 500 ppm
I20240503 08:05:35.418895 19291 webserver.cc:460] Webserver started at http://127.0.99.190:36867/ using document root <none> and password file <none>
I20240503 08:05:35.419472 19291 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:35.419528 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:35.419696 19291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:35.421300 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b1421102cfe04534873b1d6a3b682a81"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.421757 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b1421102cfe04534873b1d6a3b682a81"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.425033 19291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 08:05:35.427208 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.428216 19291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:05:35.428333 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b1421102cfe04534873b1d6a3b682a81"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.428421 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:35.441118 19291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:35.441802 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:35.441963 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:35.448683 19291 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43309
I20240503 08:05:35.448689 19357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43309 every 8 connection(s)
I20240503 08:05:35.449687 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:05:35.452423 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:35.457572 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81: Bootstrap starting.
I20240503 08:05:35.457906 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19291
I20240503 08:05:35.458027 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:05:35.460139 19358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:35.460800 19358 log.cc:824] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:35.462566 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81: No bootstrap required, opened a new log
I20240503 08:05:35.465493 19358 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } }
I20240503 08:05:35.465677 19358 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:35.465713 19358 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1421102cfe04534873b1d6a3b682a81, State: Initialized, Role: FOLLOWER
I20240503 08:05:35.466099 19358 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } }
I20240503 08:05:35.466214 19358 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:35.466282 19358 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:35.466382 19358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:35.467128 19358 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } }
I20240503 08:05:35.467440 19358 leader_election.cc:302] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1421102cfe04534873b1d6a3b682a81; no voters:
I20240503 08:05:35.467686 19358 leader_election.cc:288] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:35.468065 19363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:35.468255 19363 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [term 1 LEADER]: Becoming Leader. State: Replica: b1421102cfe04534873b1d6a3b682a81, State: Running, Role: LEADER
I20240503 08:05:35.468525 19363 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } }
I20240503 08:05:35.468731 19358 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:05:35.469882 19365 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1421102cfe04534873b1d6a3b682a81. Latest consensus state: current_term: 1 leader_uuid: "b1421102cfe04534873b1d6a3b682a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } } }
I20240503 08:05:35.469918 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1421102cfe04534873b1d6a3b682a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1421102cfe04534873b1d6a3b682a81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43309 } } }
I20240503 08:05:35.470036 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:35.469980 19365 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81 [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:35.470549 19373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:05:35.473260 19373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:05:35.477663 19373 catalog_manager.cc:1347] Generated new cluster ID: 04d51935a7344754bb6aec2648b121a6
I20240503 08:05:35.477725 19373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:05:35.490571 19373 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:05:35.491804 19373 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:05:35.503254 19373 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b1421102cfe04534873b1d6a3b682a81: Generated new TSK 0
I20240503 08:05:35.503939 19373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:05:35.518113 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:05:35.621881 19382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:35.622176 19382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:35.622258 19382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:35.625344 19382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:35.625411 19382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:35.625504 19382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:05:35.629264 19382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.19382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:35.630079 19382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:35.631194 19382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:35.639106 19390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.639820 19388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.640384 19387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:35.640972 19382 server_base.cc:1034] running on GCE node
I20240503 08:05:35.641322 19382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:35.641827 19382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:35.643003 19382 hybrid_clock.cc:648] HybridClock initialized: now 1714723535642984 us; error 34 us; skew 500 ppm
I20240503 08:05:35.644825 19382 webserver.cc:460] Webserver started at http://127.0.99.129:38665/ using document root <none> and password file <none>
I20240503 08:05:35.645346 19382 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:35.645403 19382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:35.645584 19382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:35.647245 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.647703 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.650818 19382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:05:35.652956 19396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.653892 19382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:35.653995 19382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.654081 19382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:35.673002 19382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:35.673579 19382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:35.673731 19382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:35.674250 19382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:35.675137 19382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:35.675185 19382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.675237 19382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:35.675264 19382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.683171 19382 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:44963
I20240503 08:05:35.683218 19509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:44963 every 8 connection(s)
I20240503 08:05:35.684092 19382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:05:35.692174 19510 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:35.692368 19510 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:35.692827 19510 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:35.693365 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19382
I20240503 08:05:35.693480 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:05:35.697417 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:35.726621 19322 ts_manager.cc:194] Registered new tserver with Master: b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:35.728364 19322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:45523
W20240503 08:05:35.804977 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:35.805191 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:35.805233 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:35.808188 19514 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:35.808254 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:35.808326 19514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:05:35.811830 19514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.19514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:35.812623 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:35.813624 19514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:35.821408 19522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.822127 19520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.822726 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:35.823590 19514 server_base.cc:1034] running on GCE node
I20240503 08:05:35.824218 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:35.824836 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:35.826028 19514 hybrid_clock.cc:648] HybridClock initialized: now 1714723535826017 us; error 27 us; skew 500 ppm
I20240503 08:05:35.827852 19514 webserver.cc:460] Webserver started at http://127.0.99.130:44035/ using document root <none> and password file <none>
I20240503 08:05:35.828375 19514 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:35.828425 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:35.828583 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:35.830179 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "563200d2e6694145a13a4fd97d23b4f4"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.830611 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "563200d2e6694145a13a4fd97d23b4f4"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.833799 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:05:35.835837 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.836900 19514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:35.836987 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "563200d2e6694145a13a4fd97d23b4f4"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:35.837056 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:35.853380 19514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:35.853942 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:35.854076 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:35.854550 19514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:35.855461 19514 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:35.855508 19514 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.855546 19514 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:35.855566 19514 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:35.863680 19514 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46199
I20240503 08:05:35.863706 19641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46199 every 8 connection(s)
I20240503 08:05:35.864622 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:05:35.866433 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19514
I20240503 08:05:35.866561 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:05:35.868788 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:35.873900 19642 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:35.874130 19642 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:35.874747 19642 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:35.908797 19322 ts_manager.cc:194] Registered new tserver with Master: 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199)
I20240503 08:05:35.909802 19322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:51017
W20240503 08:05:35.974813 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:35.975060 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:35.975107 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:35.978018 19645 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:35.978075 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:35.978149 19645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:05:35.981715 19645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.19645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:35.982566 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:35.983717 19645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:35.991600 19651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.992316 19654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:35.992939 19652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:35.993539 19645 server_base.cc:1034] running on GCE node
I20240503 08:05:35.993880 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:35.994400 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:35.995543 19645 hybrid_clock.cc:648] HybridClock initialized: now 1714723535995523 us; error 32 us; skew 500 ppm
I20240503 08:05:35.997366 19645 webserver.cc:460] Webserver started at http://127.0.99.131:46097/ using document root <none> and password file <none>
I20240503 08:05:35.997875 19645 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:35.997931 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:35.998101 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:35.999724 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f835c24114954b93a22708d01227b666"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:36.000171 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f835c24114954b93a22708d01227b666"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:36.003373 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:05:36.005403 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.006354 19645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:36.006449 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f835c24114954b93a22708d01227b666"
format_stamp: "Formatted at 2024-05-03 08:05:35 on dist-test-slave-11df"
I20240503 08:05:36.006525 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:36.035501 19645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:36.036101 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:36.036253 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:36.036760 19645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:36.037613 19645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:36.037665 19645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.037719 19645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:36.037742 19645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.045759 19645 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45203
I20240503 08:05:36.045784 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45203 every 8 connection(s)
I20240503 08:05:36.046686 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:05:36.054595 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19645
I20240503 08:05:36.054711 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:05:36.054903 19774 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:36.055140 19774 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:36.055732 19774 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:36.056687 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:36.087823 19322 ts_manager.cc:194] Registered new tserver with Master: f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:36.088521 19322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55585
W20240503 08:05:36.159257 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:36.159452 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:36.159508 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:36.162427 19777 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:36.162488 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:36.162569 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:36.165997 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.19777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:36.166834 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:36.168010 19777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:36.176828 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:36.177098 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:36.177860 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:36.178622 19777 server_base.cc:1034] running on GCE node
I20240503 08:05:36.179041 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:36.179577 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:36.180727 19777 hybrid_clock.cc:648] HybridClock initialized: now 1714723536180710 us; error 28 us; skew 500 ppm
I20240503 08:05:36.182520 19777 webserver.cc:460] Webserver started at http://127.0.99.132:34321/ using document root <none> and password file <none>
I20240503 08:05:36.183084 19777 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:36.183138 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:36.183310 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:36.184911 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ea59580caefd4780bb4f019a0133e704"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.185365 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ea59580caefd4780bb4f019a0133e704"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.188539 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 08:05:36.190609 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.191605 19777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:05:36.191690 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ea59580caefd4780bb4f019a0133e704"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.191759 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:36.205320 19777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:36.205899 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:36.206043 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:36.206521 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:36.207428 19777 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:36.207474 19777 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.207520 19777 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:36.207540 19777 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.215441 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37101
I20240503 08:05:36.215468 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37101 every 8 connection(s)
I20240503 08:05:36.216352 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:36.222107 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19777
I20240503 08:05:36.222221 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:05:36.224418 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:36.224800 19906 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:36.224998 19906 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:36.225507 19906 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:36.261548 19322 ts_manager.cc:194] Registered new tserver with Master: ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101)
I20240503 08:05:36.262243 19322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:54981
W20240503 08:05:36.327304 19909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:36.327499 19909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:36.327546 19909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:36.330452 19909 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:36.330511 19909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:36.330590 19909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:05:36.334101 19909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.19909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:36.334889 19909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:36.335970 19909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:36.345558 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:36.345800 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:36.346170 19909 server_base.cc:1034] running on GCE node
W20240503 08:05:36.346655 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:36.347049 19909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:36.347589 19909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:36.348726 19909 hybrid_clock.cc:648] HybridClock initialized: now 1714723536348708 us; error 36 us; skew 500 ppm
I20240503 08:05:36.350551 19909 webserver.cc:460] Webserver started at http://127.0.99.133:34469/ using document root <none> and password file <none>
I20240503 08:05:36.351109 19909 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:36.351166 19909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:36.351341 19909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:36.352908 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2d709df031144ec6b5c0fd04315bd1fb"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.353356 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2d709df031144ec6b5c0fd04315bd1fb"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.356458 19909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:05:36.358466 19924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.359390 19909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:36.359484 19909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2d709df031144ec6b5c0fd04315bd1fb"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:36.359556 19909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:36.373617 19909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:36.374174 19909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:36.374323 19909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:36.374912 19909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:36.375819 19909 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:36.375866 19909 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.375916 19909 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:36.375943 19909 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:36.383512 19909 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44637
I20240503 08:05:36.383553 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44637 every 8 connection(s)
I20240503 08:05:36.384399 19909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:05:36.389741 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 19909
I20240503 08:05:36.389858 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:05:36.392776 20038 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:36.392971 20038 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:36.393463 20038 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:36.426913 19322 ts_manager.cc:194] Registered new tserver with Master: 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637)
I20240503 08:05:36.427700 19322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:37137
I20240503 08:05:36.428037 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:05:36.436604 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:05:36.436679 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19909
I20240503 08:05:36.441962 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19777
I20240503 08:05:36.731277 19510 heartbeater.cc:503] Master 127.0.99.190:43309 was elected leader, sending a full tablet report...
I20240503 08:05:36.912112 19642 heartbeater.cc:503] Master 127.0.99.190:43309 was elected leader, sending a full tablet report...
I20240503 08:05:37.090721 19774 heartbeater.cc:503] Master 127.0.99.190:43309 was elected leader, sending a full tablet report...
I20240503 08:05:40.197805 19322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57162:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:40.200873 19322 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:40.217586 19444 tablet_service.cc:1467] Processing CreateTablet for tablet 40ea96d08bcb46028cc7f907b489e76b (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.217586 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 40ea96d08bcb46028cc7f907b489e76b (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.217586 19576 tablet_service.cc:1467] Processing CreateTablet for tablet 40ea96d08bcb46028cc7f907b489e76b (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.218101 19706 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7a4621fd0462c9feeb0f454e44d03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.217736 19575 tablet_service.cc:1467] Processing CreateTablet for tablet 54c2f20ffc844500adc57b6961b3e9d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.218438 19574 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7a4621fd0462c9feeb0f454e44d03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.218549 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ea96d08bcb46028cc7f907b489e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.218549 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ea96d08bcb46028cc7f907b489e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.218549 19444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ea96d08bcb46028cc7f907b489e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.217586 19707 tablet_service.cc:1467] Processing CreateTablet for tablet 54c2f20ffc844500adc57b6961b3e9d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.218986 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c2f20ffc844500adc57b6961b3e9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.219645 19575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c2f20ffc844500adc57b6961b3e9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.219997 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7a4621fd0462c9feeb0f454e44d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.220093 19443 tablet_service.cc:1467] Processing CreateTablet for tablet 54c2f20ffc844500adc57b6961b3e9d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.220351 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7a4621fd0462c9feeb0f454e44d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.220453 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7a4621fd0462c9feeb0f454e44d03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:40.221908 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c2f20ffc844500adc57b6961b3e9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.222309 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7a4621fd0462c9feeb0f454e44d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:40.225700 20057 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:40.226022 20058 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:40.227936 20058 tablet_bootstrap.cc:654] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.227995 20057 tablet_bootstrap.cc:654] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.228649 20058 log.cc:824] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:40.228750 20057 log.cc:824] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:40.229590 20059 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:40.230511 20058 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:40.230670 20058 ts_tablet_manager.cc:1367] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:05:40.231495 20059 tablet_bootstrap.cc:654] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.231736 20057 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:40.231881 20057 ts_tablet_manager.cc:1367] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:05:40.232219 20059 log.cc:824] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:40.233764 20058 raft_consensus.cc:348] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.233893 20059 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:40.233951 20058 raft_consensus.cc:374] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.234021 20058 raft_consensus.cc:729] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.234038 20059 ts_tablet_manager.cc:1367] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:05:40.234433 20058 consensus_queue.cc:260] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.235253 20058 ts_tablet_manager.cc:1398] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:05:40.235584 20058 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:40.235538 20057 raft_consensus.cc:348] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.235810 20057 raft_consensus.cc:374] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.235896 20057 raft_consensus.cc:729] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.236397 20057 consensus_queue.cc:260] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.236516 20058 tablet_bootstrap.cc:654] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.237453 20059 raft_consensus.cc:348] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.237634 20059 raft_consensus.cc:374] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.237684 20059 raft_consensus.cc:729] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.238101 20057 ts_tablet_manager.cc:1398] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 08:05:40.238046 20059 consensus_queue.cc:260] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.238384 20057 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:40.239154 20058 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:40.239239 20058 ts_tablet_manager.cc:1367] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:05:40.239436 20057 tablet_bootstrap.cc:654] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.239761 20059 ts_tablet_manager.cc:1398] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 08:05:40.239745 20058 raft_consensus.cc:348] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.239831 20058 raft_consensus.cc:374] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.239868 20058 raft_consensus.cc:729] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.239975 20058 consensus_queue.cc:260] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.240090 20059 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:40.240236 20058 ts_tablet_manager.cc:1398] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:40.240411 20058 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:40.240733 20057 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:40.240801 20057 ts_tablet_manager.cc:1367] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:05:40.241016 20059 tablet_bootstrap.cc:654] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.241245 20058 tablet_bootstrap.cc:654] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.241278 20057 raft_consensus.cc:348] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.241365 20057 raft_consensus.cc:374] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.241394 20057 raft_consensus.cc:729] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.241501 20057 consensus_queue.cc:260] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.241776 20057 ts_tablet_manager.cc:1398] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:40.241928 20057 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:40.242197 20059 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:40.242287 20059 ts_tablet_manager.cc:1367] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:40.242688 20057 tablet_bootstrap.cc:654] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.242741 20059 raft_consensus.cc:348] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.242834 20059 raft_consensus.cc:374] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.242864 20059 raft_consensus.cc:729] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.242993 20059 consensus_queue.cc:260] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.243332 20059 ts_tablet_manager.cc:1398] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:40.243481 20059 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:40.244264 20059 tablet_bootstrap.cc:654] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:40.245285 20058 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:40.245296 20057 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:40.245358 20058 ts_tablet_manager.cc:1367] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:05:40.245366 20057 ts_tablet_manager.cc:1367] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:40.245743 20058 raft_consensus.cc:348] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.245826 20058 raft_consensus.cc:374] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.245779 20057 raft_consensus.cc:348] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.245858 20058 raft_consensus.cc:729] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.245874 20057 raft_consensus.cc:374] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.245905 20057 raft_consensus.cc:729] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.245963 20058 consensus_queue.cc:260] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.246003 20057 consensus_queue.cc:260] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.246203 20058 ts_tablet_manager.cc:1398] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:40.246223 20057 ts_tablet_manager.cc:1398] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:40.246379 20059 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:40.246454 20059 ts_tablet_manager.cc:1367] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:40.246865 20059 raft_consensus.cc:348] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.246987 20059 raft_consensus.cc:374] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:40.247031 20059 raft_consensus.cc:729] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:40.247139 20059 consensus_queue.cc:260] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.247363 20059 ts_tablet_manager.cc:1398] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 08:05:40.300215 19775 tablet.cc:2369] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:05:40.368604 19643 tablet.cc:2369] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:05:40.401176 20070 raft_consensus.cc:482] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:40.401379 20070 raft_consensus.cc:504] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.402513 20070 leader_election.cc:288] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:40.406772 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c2f20ffc844500adc57b6961b3e9d5" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:40.406724 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c2f20ffc844500adc57b6961b3e9d5" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:40.407096 19596 raft_consensus.cc:2454] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:40.407096 19728 raft_consensus.cc:2454] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:40.407495 19397 leader_election.cc:302] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:40.407714 20070 raft_consensus.cc:2781] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:40.407785 20070 raft_consensus.cc:482] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:40.407814 20070 raft_consensus.cc:3037] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.408653 20070 raft_consensus.cc:504] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.408995 20070 leader_election.cc:288] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:40.409283 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c2f20ffc844500adc57b6961b3e9d5" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:40.409296 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c2f20ffc844500adc57b6961b3e9d5" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:40.409395 19728 raft_consensus.cc:3037] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.409404 19596 raft_consensus.cc:3037] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.410295 19596 raft_consensus.cc:2454] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:40.410559 19398 leader_election.cc:302] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:40.410607 19728 raft_consensus.cc:2454] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:40.410729 20070 raft_consensus.cc:2781] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:40.410815 20070 raft_consensus.cc:686] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Leader. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:40.411135 20070 consensus_queue.cc:237] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.414117 19322 catalog_manager.cc:5484] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: term changed from 0 to 1, leader changed from <none> to b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:40.438097 19511 tablet.cc:2369] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:05:40.587656 20064 raft_consensus.cc:482] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:40.587793 20064 raft_consensus.cc:504] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.588780 20064 leader_election.cc:288] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:40.592656 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ea96d08bcb46028cc7f907b489e76b" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:40.592816 19596 raft_consensus.cc:2454] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 0.
I20240503 08:05:40.592797 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ea96d08bcb46028cc7f907b489e76b" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" is_pre_election: true
I20240503 08:05:40.593001 19464 raft_consensus.cc:2454] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 0.
I20240503 08:05:40.593173 19662 leader_election.cc:302] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:40.593410 20064 raft_consensus.cc:2781] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:40.593472 20064 raft_consensus.cc:482] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:40.593518 20064 raft_consensus.cc:3037] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.594336 20064 raft_consensus.cc:504] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.594659 20064 leader_election.cc:288] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:40.595014 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ea96d08bcb46028cc7f907b489e76b" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:40.595018 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ea96d08bcb46028cc7f907b489e76b" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:40.595118 19464 raft_consensus.cc:3037] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.595151 19596 raft_consensus.cc:3037] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.595890 19596 raft_consensus.cc:2454] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 1.
I20240503 08:05:40.595896 19464 raft_consensus.cc:2454] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 1.
I20240503 08:05:40.596158 19662 leader_election.cc:302] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:40.596319 20064 raft_consensus.cc:2781] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:40.596532 20064 raft_consensus.cc:686] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Becoming Leader. State: Replica: f835c24114954b93a22708d01227b666, State: Running, Role: LEADER
I20240503 08:05:40.596822 20064 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } }
I20240503 08:05:40.599650 19322 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 reported cstate change: term changed from 0 to 1, leader changed from <none> to f835c24114954b93a22708d01227b666 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:40.658711 20063 raft_consensus.cc:482] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:40.658862 20063 raft_consensus.cc:504] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.659950 20063 leader_election.cc:288] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:40.664368 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7a4621fd0462c9feeb0f454e44d03" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" is_pre_election: true
I20240503 08:05:40.664418 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7a4621fd0462c9feeb0f454e44d03" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:40.664517 19464 raft_consensus.cc:2454] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:40.664552 19728 raft_consensus.cc:2454] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:40.664866 19530 leader_election.cc:302] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:40.665129 20063 raft_consensus.cc:2781] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:40.665191 20063 raft_consensus.cc:482] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:40.665225 20063 raft_consensus.cc:3037] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.666015 20063 raft_consensus.cc:504] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.666375 20063 leader_election.cc:288] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Requested vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:40.666743 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7a4621fd0462c9feeb0f454e44d03" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:40.666801 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7a4621fd0462c9feeb0f454e44d03" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:40.666857 19728 raft_consensus.cc:3037] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.666949 19464 raft_consensus.cc:3037] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:40.667578 19728 raft_consensus.cc:2454] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:40.667616 19464 raft_consensus.cc:2454] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:40.667893 19529 leader_election.cc:302] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:40.668057 20063 raft_consensus.cc:2781] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:40.668257 20063 raft_consensus.cc:686] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Becoming Leader. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Running, Role: LEADER
I20240503 08:05:40.668532 20063 consensus_queue.cc:237] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:40.671486 19322 catalog_manager.cc:5484] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:40.866160 20070 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:40.870935 20066 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:05:40.886186 20084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:40.886458 20084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:40.889667 20084 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:05:40.982662 19464 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:40.983094 19728 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:40.983469 20063 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:40.983711 20082 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:40.987066 19463 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:40.987638 20064 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:40.989168 19596 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:40.989508 20064 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.004490 20086 mvcc.cc:205] Tried to move back new op lower bound from 7023507623854178304 to 7023507622580957184. Current Snapshot: MvccSnapshot[applied={T|T < 7023507623854178304}]
I20240503 08:05:41.004686 20087 mvcc.cc:205] Tried to move back new op lower bound from 7023507623914749952 to 7023507622286888960. Current Snapshot: MvccSnapshot[applied={T|T < 7023507623876063232}]
I20240503 08:05:41.004781 20130 mvcc.cc:205] Tried to move back new op lower bound from 7023507623876063232 to 7023507622286888960. Current Snapshot: MvccSnapshot[applied={T|T < 7023507623876063232}]
INSERT report
rows total: 2000
time total: 114.778 ms
time per row: 0.0573892 ms
I20240503 08:05:41.045702 19321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57162:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:41.046216 19321 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:41.051414 19444 tablet_service.cc:1467] Processing CreateTablet for tablet f8530aaa9e8a43e58070765927a75798 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.051723 19444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8530aaa9e8a43e58070765927a75798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.051823 19707 tablet_service.cc:1467] Processing CreateTablet for tablet f8530aaa9e8a43e58070765927a75798 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.052129 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8530aaa9e8a43e58070765927a75798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.053933 20140 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.054352 19442 tablet_service.cc:1467] Processing CreateTablet for tablet bc872d506be04591a178b33ca06d09f2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.054546 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc872d506be04591a178b33ca06d09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.054950 19708 tablet_service.cc:1467] Processing CreateTablet for tablet bc872d506be04591a178b33ca06d09f2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.055038 20140 tablet_bootstrap.cc:654] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.055222 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc872d506be04591a178b33ca06d09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.055321 19444 tablet_service.cc:1467] Processing CreateTablet for tablet eb2dde0bfa18445d8b4a05db1030ef65 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.055521 19444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2dde0bfa18445d8b4a05db1030ef65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.055475 19574 tablet_service.cc:1467] Processing CreateTablet for tablet eb2dde0bfa18445d8b4a05db1030ef65 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.055712 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2dde0bfa18445d8b4a05db1030ef65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.057332 19576 tablet_service.cc:1467] Processing CreateTablet for tablet f8530aaa9e8a43e58070765927a75798 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.057557 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8530aaa9e8a43e58070765927a75798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.057757 19707 tablet_service.cc:1467] Processing CreateTablet for tablet eb2dde0bfa18445d8b4a05db1030ef65 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.057843 20143 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.058002 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2dde0bfa18445d8b4a05db1030ef65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.057873 19575 tablet_service.cc:1467] Processing CreateTablet for tablet bc872d506be04591a178b33ca06d09f2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.058315 19575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc872d506be04591a178b33ca06d09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.059090 20143 tablet_bootstrap.cc:654] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.059442 20141 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.060184 20141 tablet_bootstrap.cc:654] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.061314 20143 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.061403 20143 ts_tablet_manager.cc:1367] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:05:41.062137 20141 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.062239 20141 ts_tablet_manager.cc:1367] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:41.062398 20140 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.062490 20140 ts_tablet_manager.cc:1367] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:05:41.062661 20141 raft_consensus.cc:348] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.062744 20141 raft_consensus.cc:374] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.062774 20141 raft_consensus.cc:729] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.062898 20141 consensus_queue.cc:260] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.063205 20141 ts_tablet_manager.cc:1398] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.063369 20141 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.064036 20141 tablet_bootstrap.cc:654] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.064904 20141 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.064975 20141 ts_tablet_manager.cc:1367] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:41.065295 20141 raft_consensus.cc:348] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.065373 20141 raft_consensus.cc:374] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.065402 20141 raft_consensus.cc:729] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.065515 20141 consensus_queue.cc:260] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.065806 20141 ts_tablet_manager.cc:1398] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.065943 20141 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.066581 20141 tablet_bootstrap.cc:654] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.062986 20140 raft_consensus.cc:348] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.067070 20140 raft_consensus.cc:374] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.067117 20140 raft_consensus.cc:729] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.067261 20140 consensus_queue.cc:260] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.067516 20140 ts_tablet_manager.cc:1398] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:05:41.067664 20140 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.067782 20141 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.067855 20141 ts_tablet_manager.cc:1367] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:05:41.068521 20140 tablet_bootstrap.cc:654] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.068310 20141 raft_consensus.cc:348] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.068668 20141 raft_consensus.cc:374] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.068717 20141 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.068853 20141 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.061902 20143 raft_consensus.cc:348] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.069077 20143 raft_consensus.cc:374] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.069126 20143 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.069186 20141 ts_tablet_manager.cc:1398] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:41.069258 20143 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.069550 20143 ts_tablet_manager.cc:1398] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:05:41.069766 20143 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.070619 20143 tablet_bootstrap.cc:654] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.071609 20143 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.071678 20143 ts_tablet_manager.cc:1367] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:41.072033 20143 raft_consensus.cc:348] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.072122 20143 raft_consensus.cc:374] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.072149 20143 raft_consensus.cc:729] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.072260 20143 consensus_queue.cc:260] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.072590 20143 ts_tablet_manager.cc:1398] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.072739 20143 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.073549 20143 tablet_bootstrap.cc:654] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.074463 20143 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.074535 20143 ts_tablet_manager.cc:1367] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:41.074851 20143 raft_consensus.cc:348] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.074945 20143 raft_consensus.cc:374] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.074980 20143 raft_consensus.cc:729] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.075084 20143 consensus_queue.cc:260] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.075294 20143 ts_tablet_manager.cc:1398] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:41.076690 20140 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.076788 20140 ts_tablet_manager.cc:1367] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 08:05:41.077284 20140 raft_consensus.cc:348] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.077405 20140 raft_consensus.cc:374] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.077454 20140 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.077587 20140 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.077881 20140 ts_tablet_manager.cc:1398] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:41.078073 20140 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.079419 20140 tablet_bootstrap.cc:654] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.080478 20140 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.080545 20140 ts_tablet_manager.cc:1367] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:05:41.080884 20140 raft_consensus.cc:348] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.080962 20140 raft_consensus.cc:374] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.080996 20140 raft_consensus.cc:729] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.081104 20140 consensus_queue.cc:260] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.081321 20140 ts_tablet_manager.cc:1398] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:41.085372 20063 raft_consensus.cc:482] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.085456 20063 raft_consensus.cc:504] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.085774 20063 leader_election.cc:288] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.086081 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc872d506be04591a178b33ca06d09f2" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" is_pre_election: true
I20240503 08:05:41.086174 19464 raft_consensus.cc:2454] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:41.086174 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc872d506be04591a178b33ca06d09f2" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:41.086308 19728 raft_consensus.cc:2454] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:41.086412 19530 leader_election.cc:302] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.086589 20063 raft_consensus.cc:2781] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.086650 20063 raft_consensus.cc:482] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.086694 20063 raft_consensus.cc:3037] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.087484 20063 raft_consensus.cc:504] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.087771 20063 leader_election.cc:288] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Requested vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.088043 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc872d506be04591a178b33ca06d09f2" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:41.088132 19464 raft_consensus.cc:3037] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.088150 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc872d506be04591a178b33ca06d09f2" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:41.088263 19728 raft_consensus.cc:3037] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.088838 19464 raft_consensus.cc:2454] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:41.088994 19728 raft_consensus.cc:2454] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:41.089094 19530 leader_election.cc:302] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.089349 20063 raft_consensus.cc:2781] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.089430 20063 raft_consensus.cc:686] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Becoming Leader. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Running, Role: LEADER
I20240503 08:05:41.089550 20063 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.090978 19320 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:41.094841 20119 raft_consensus.cc:482] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.094993 20119 raft_consensus.cc:504] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.095376 20119 leader_election.cc:288] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.095675 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:41.095734 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:41.095780 19596 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.095865 19728 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.096033 19398 leader_election.cc:302] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.096239 20119 raft_consensus.cc:2781] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.096282 20119 raft_consensus.cc:482] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.096315 20119 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.097081 20119 raft_consensus.cc:504] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.097352 20119 leader_election.cc:288] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.097623 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:41.097714 19596 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.097733 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:41.097836 19728 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.098486 19596 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.098595 19728 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.098737 19398 leader_election.cc:302] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.098891 20119 raft_consensus.cc:2781] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.098989 20119 raft_consensus.cc:686] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Leader. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:41.099114 20119 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.100574 19320 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: term changed from 0 to 1, leader changed from <none> to b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:41.173609 20119 raft_consensus.cc:482] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.173763 20119 raft_consensus.cc:504] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.174192 20119 leader_election.cc:288] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.174665 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:41.174667 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:41.174793 19728 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.174821 19596 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.175102 19397 leader_election.cc:302] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.175276 20119 raft_consensus.cc:2781] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.175318 20119 raft_consensus.cc:482] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.175343 20119 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.176148 20119 raft_consensus.cc:504] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.176472 20119 leader_election.cc:288] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.176807 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:41.176880 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:41.176919 19596 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.177033 19728 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.177651 19596 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.177753 19728 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.177917 19398 leader_election.cc:302] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.178064 20119 raft_consensus.cc:2781] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.178170 20119 raft_consensus.cc:686] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Leader. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:41.178333 20119 consensus_queue.cc:237] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.179967 19320 catalog_manager.cc:5484] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: term changed from 0 to 1, leader changed from <none> to b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:41.313808 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:41.314030 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:41.316973 20155 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:05:41.385802 19596 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.386178 19728 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.386611 19728 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.386986 20070 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.387063 20119 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.387598 19596 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.387580 20119 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.391736 20070 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.401657 19464 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.401667 19728 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:41.402057 20063 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.402189 20082 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 92.5007 ms
time per row: 0.0462503 ms
I20240503 08:05:41.450014 19320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57162:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:41.450500 19320 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:41.455636 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 08dc753bd09c48d4baa70f08b643cfec (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.455724 19707 tablet_service.cc:1467] Processing CreateTablet for tablet 08dc753bd09c48d4baa70f08b643cfec (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.455955 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dc753bd09c48d4baa70f08b643cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.456015 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dc753bd09c48d4baa70f08b643cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.456461 19575 tablet_service.cc:1467] Processing CreateTablet for tablet 08dc753bd09c48d4baa70f08b643cfec (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.456523 19576 tablet_service.cc:1467] Processing CreateTablet for tablet 170ea8d950d4437eb5c3bdae4956741c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.456688 19575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dc753bd09c48d4baa70f08b643cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.456928 19574 tablet_service.cc:1467] Processing CreateTablet for tablet a8cd2c0cf0a04ccfb83d98aefa18e188 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.457134 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cd2c0cf0a04ccfb83d98aefa18e188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.458114 20141 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.458565 19444 tablet_service.cc:1467] Processing CreateTablet for tablet a8cd2c0cf0a04ccfb83d98aefa18e188 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.458778 19444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cd2c0cf0a04ccfb83d98aefa18e188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.458981 20141 tablet_bootstrap.cc:654] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.459031 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ea8d950d4437eb5c3bdae4956741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.459087 20143 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.459851 20143 tablet_bootstrap.cc:654] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.459939 20140 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.460700 20140 tablet_bootstrap.cc:654] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.461635 20141 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.461724 20141 ts_tablet_manager.cc:1367] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:05:41.461689 19708 tablet_service.cc:1467] Processing CreateTablet for tablet a8cd2c0cf0a04ccfb83d98aefa18e188 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.461918 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cd2c0cf0a04ccfb83d98aefa18e188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.462193 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 170ea8d950d4437eb5c3bdae4956741c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.462168 20141 raft_consensus.cc:348] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.462270 20141 raft_consensus.cc:374] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.462297 20141 raft_consensus.cc:729] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.462390 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ea8d950d4437eb5c3bdae4956741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.462430 20141 consensus_queue.cc:260] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.462687 20141 ts_tablet_manager.cc:1398] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:41.463584 20141 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.464223 19707 tablet_service.cc:1467] Processing CreateTablet for tablet 170ea8d950d4437eb5c3bdae4956741c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:41.464473 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ea8d950d4437eb5c3bdae4956741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:41.464871 20141 tablet_bootstrap.cc:654] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.465996 20140 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.466091 20140 ts_tablet_manager.cc:1367] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:05:41.466504 20140 raft_consensus.cc:348] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.466601 20140 raft_consensus.cc:374] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.466629 20140 raft_consensus.cc:729] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.466742 20140 consensus_queue.cc:260] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.467008 20140 ts_tablet_manager.cc:1398] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.467217 20140 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.468063 20140 tablet_bootstrap.cc:654] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.469241 20140 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.469322 20140 ts_tablet_manager.cc:1367] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:41.469797 20140 raft_consensus.cc:348] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.469897 20140 raft_consensus.cc:374] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.469923 20140 raft_consensus.cc:729] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.470114 20140 consensus_queue.cc:260] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.470355 20140 ts_tablet_manager.cc:1398] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.470495 20140 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:41.471333 20140 tablet_bootstrap.cc:654] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.472596 20140 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad: No bootstrap required, opened a new log
I20240503 08:05:41.472718 20140 ts_tablet_manager.cc:1367] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:41.473117 20140 raft_consensus.cc:348] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.473213 20140 raft_consensus.cc:374] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.473286 20140 raft_consensus.cc:729] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.473434 20140 consensus_queue.cc:260] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.473703 20140 ts_tablet_manager.cc:1398] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.474198 20143 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.474318 20143 ts_tablet_manager.cc:1367] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240503 08:05:41.474318 20141 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.474402 20141 ts_tablet_manager.cc:1367] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:05:41.474766 20143 raft_consensus.cc:348] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.474835 20141 raft_consensus.cc:348] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.474890 20143 raft_consensus.cc:374] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.474951 20141 raft_consensus.cc:374] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.474967 20143 raft_consensus.cc:729] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.474982 20141 raft_consensus.cc:729] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.475106 20141 consensus_queue.cc:260] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.475101 20143 consensus_queue.cc:260] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.475358 20141 ts_tablet_manager.cc:1398] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.475373 20143 ts_tablet_manager.cc:1398] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.475504 20143 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.476363 20143 tablet_bootstrap.cc:654] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.476579 20141 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Bootstrap starting.
I20240503 08:05:41.478138 20141 tablet_bootstrap.cc:654] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.478351 20143 tablet_bootstrap.cc:492] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.478432 20143 ts_tablet_manager.cc:1367] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:41.478813 20143 raft_consensus.cc:348] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.478883 20078 raft_consensus.cc:482] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.478950 20143 raft_consensus.cc:374] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.478976 20078 raft_consensus.cc:504] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.478994 20143 raft_consensus.cc:729] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.479297 20143 consensus_queue.cc:260] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.479410 20078 leader_election.cc:288] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:41.479609 20143 ts_tablet_manager.cc:1398] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:41.479758 20143 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:41.480036 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cd2c0cf0a04ccfb83d98aefa18e188" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" is_pre_election: true
I20240503 08:05:41.480038 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cd2c0cf0a04ccfb83d98aefa18e188" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:41.480163 19464 raft_consensus.cc:2454] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 0.
I20240503 08:05:41.480178 19595 raft_consensus.cc:2454] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 0.
I20240503 08:05:41.480455 19662 leader_election.cc:302] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.480595 20078 raft_consensus.cc:2781] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.480643 20078 raft_consensus.cc:482] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.480674 20078 raft_consensus.cc:3037] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.481199 20143 tablet_bootstrap.cc:654] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:41.481652 20078 raft_consensus.cc:504] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.481945 20078 leader_election.cc:288] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:41.482275 20143 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: No bootstrap required, opened a new log
I20240503 08:05:41.482228 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cd2c0cf0a04ccfb83d98aefa18e188" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:41.482254 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cd2c0cf0a04ccfb83d98aefa18e188" candidate_uuid: "f835c24114954b93a22708d01227b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:41.482352 19464 raft_consensus.cc:3037] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.482344 19595 raft_consensus.cc:3037] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.482364 20141 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: No bootstrap required, opened a new log
I20240503 08:05:41.482350 20143 ts_tablet_manager.cc:1367] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:05:41.482430 20141 ts_tablet_manager.cc:1367] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:05:41.482757 20143 raft_consensus.cc:348] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.482848 20143 raft_consensus.cc:374] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.482815 20141 raft_consensus.cc:348] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.482877 20143 raft_consensus.cc:729] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.482896 20141 raft_consensus.cc:374] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:41.482952 20141 raft_consensus.cc:729] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835c24114954b93a22708d01227b666, State: Initialized, Role: FOLLOWER
I20240503 08:05:41.483037 20143 consensus_queue.cc:260] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.483062 20141 consensus_queue.cc:260] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.483332 19464 raft_consensus.cc:2454] T a8cd2c0cf0a04ccfb83d98aefa18e188 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 1.
I20240503 08:05:41.483376 20141 ts_tablet_manager.cc:1398] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.483583 19662 leader_election.cc:302] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.483597 19595 raft_consensus.cc:2454] T a8cd2c0cf0a04ccfb83d98aefa18e188 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835c24114954b93a22708d01227b666 in term 1.
I20240503 08:05:41.483712 20078 raft_consensus.cc:2781] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.483793 20078 raft_consensus.cc:686] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Becoming Leader. State: Replica: f835c24114954b93a22708d01227b666, State: Running, Role: LEADER
I20240503 08:05:41.483879 20143 ts_tablet_manager.cc:1398] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:41.483898 20078 consensus_queue.cc:237] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.485939 19320 catalog_manager.cc:5484] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 reported cstate change: term changed from 0 to 1, leader changed from <none> to f835c24114954b93a22708d01227b666 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } }
I20240503 08:05:41.491046 20063 raft_consensus.cc:482] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.491147 20063 raft_consensus.cc:504] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.491441 20063 leader_election.cc:288] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.491752 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ea8d950d4437eb5c3bdae4956741c" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:41.491745 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ea8d950d4437eb5c3bdae4956741c" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" is_pre_election: true
I20240503 08:05:41.491866 19728 raft_consensus.cc:2454] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:41.491868 19464 raft_consensus.cc:2454] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 0.
I20240503 08:05:41.492127 19529 leader_election.cc:302] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.492285 20063 raft_consensus.cc:2781] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.492337 20063 raft_consensus.cc:482] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.492367 20063 raft_consensus.cc:3037] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.493007 20063 raft_consensus.cc:504] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.493297 20063 leader_election.cc:288] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Requested vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.493597 19464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ea8d950d4437eb5c3bdae4956741c" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:41.493642 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170ea8d950d4437eb5c3bdae4956741c" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:41.493708 19464 raft_consensus.cc:3037] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.493762 19728 raft_consensus.cc:3037] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.494563 19728 raft_consensus.cc:2454] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:41.494565 19464 raft_consensus.cc:2454] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 1.
I20240503 08:05:41.494822 19529 leader_election.cc:302] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.494983 20063 raft_consensus.cc:2781] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.495082 20063 raft_consensus.cc:686] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Becoming Leader. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Running, Role: LEADER
I20240503 08:05:41.495234 20063 consensus_queue.cc:237] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.496821 19320 catalog_manager.cc:5484] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:41.788277 20187 raft_consensus.cc:482] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:41.788445 20187 raft_consensus.cc:504] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.788949 20187 leader_election.cc:288] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.789407 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dc753bd09c48d4baa70f08b643cfec" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4" is_pre_election: true
I20240503 08:05:41.789433 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dc753bd09c48d4baa70f08b643cfec" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666" is_pre_election: true
I20240503 08:05:41.789564 19595 raft_consensus.cc:2454] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.789588 19728 raft_consensus.cc:2454] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 0.
I20240503 08:05:41.789878 19398 leader_election.cc:302] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad; no voters:
I20240503 08:05:41.790064 20187 raft_consensus.cc:2781] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:41.790114 20187 raft_consensus.cc:482] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:41.790151 20187 raft_consensus.cc:3037] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.790973 20187 raft_consensus.cc:504] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.791291 20187 leader_election.cc:288] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Requested vote from peers 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:41.791628 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dc753bd09c48d4baa70f08b643cfec" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:41.791746 19728 raft_consensus.cc:3037] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.791704 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08dc753bd09c48d4baa70f08b643cfec" candidate_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:41.791837 19595 raft_consensus.cc:3037] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:41.792460 19728 raft_consensus.cc:2454] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.792534 19595 raft_consensus.cc:2454] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38441293bcc4b57b5f4c09adeaf70ad in term 1.
I20240503 08:05:41.792711 19397 leader_election.cc:302] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:41.792937 20187 raft_consensus.cc:2781] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:41.793030 20187 raft_consensus.cc:686] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Leader. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:41.793172 20187 consensus_queue.cc:237] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } }
I20240503 08:05:41.794842 19320 catalog_manager.cc:5484] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: term changed from 0 to 1, leader changed from <none> to b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:41.928504 20203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:41.928752 20203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:41.931854 20203 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:05:41.939447 20063 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:41.942806 20190 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:05:41.995049 20078 consensus_queue.cc:1035] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:42.005259 20078 consensus_queue.cc:1035] T a8cd2c0cf0a04ccfb83d98aefa18e188 P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:42.025449 19595 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:42.025853 20187 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:42.032069 19728 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:42.032491 20184 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.94 ms
time per row: 0.0509702 ms
I20240503 08:05:42.076828 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:44637
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:43309
--webserver_port=34469
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:05:42.180276 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:42.180490 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:42.180532 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:42.183566 20266 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:42.183629 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:42.183694 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:05:42.187305 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:44637
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=34469
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.20266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:42.188146 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:42.189190 20266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:42.197854 20272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:42.198105 20274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:42.198879 20266 server_base.cc:1034] running on GCE node
W20240503 08:05:42.199402 20271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:42.199797 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:42.200337 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:42.201481 20266 hybrid_clock.cc:648] HybridClock initialized: now 1714723542201469 us; error 28 us; skew 500 ppm
I20240503 08:05:42.203397 20266 webserver.cc:460] Webserver started at http://127.0.99.133:34469/ using document root <none> and password file <none>
I20240503 08:05:42.203958 20266 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:42.204041 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:42.207473 20266 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:05:42.209339 20280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.210325 20266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:42.210423 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2d709df031144ec6b5c0fd04315bd1fb"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:42.210806 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:42.229336 20266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:42.229965 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:42.230122 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:42.230657 20266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:42.231564 20266 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:42.231611 20266 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.231662 20266 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:42.231688 20266 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.240095 20266 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44637
I20240503 08:05:42.240231 20393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44637 every 8 connection(s)
I20240503 08:05:42.241128 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:05:42.242080 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 20266
I20240503 08:05:42.242519 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:37101
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:43309
--webserver_port=34321
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:42.250386 20394 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:42.250653 20394 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:42.251381 20394 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:42.252681 19316 ts_manager.cc:194] Re-registered known tserver with Master: 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637)
I20240503 08:05:42.253265 19316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:46493
W20240503 08:05:42.349682 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:42.349874 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:42.349921 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:42.352897 20397 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:42.352957 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:42.353051 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:42.356532 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:37101
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=34321
--tserver_master_addrs=127.0.99.190:43309
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:42.357412 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:42.358532 20397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:42.366847 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:42.367120 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:42.367846 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:42.368494 20397 server_base.cc:1034] running on GCE node
I20240503 08:05:42.368842 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:42.369361 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:42.370520 20397 hybrid_clock.cc:648] HybridClock initialized: now 1714723542370498 us; error 37 us; skew 500 ppm
I20240503 08:05:42.372376 20397 webserver.cc:460] Webserver started at http://127.0.99.132:34321/ using document root <none> and password file <none>
I20240503 08:05:42.372993 20397 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:42.373064 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:42.376235 20397 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:05:42.378062 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.379110 20397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:42.379213 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ea59580caefd4780bb4f019a0133e704"
format_stamp: "Formatted at 2024-05-03 08:05:36 on dist-test-slave-11df"
I20240503 08:05:42.379648 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:42.403767 20397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:42.404371 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:42.404522 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:42.405063 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:42.405942 20397 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:42.405987 20397 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.406033 20397 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:42.406060 20397 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:42.414000 20397 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37101
I20240503 08:05:42.414070 20524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37101 every 8 connection(s)
I20240503 08:05:42.415076 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:42.418797 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 20397
I20240503 08:05:42.424311 20525 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43309
I20240503 08:05:42.424511 20525 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:42.425012 20525 heartbeater.cc:511] Master 127.0.99.190:43309 requested a full tablet report, sending...
I20240503 08:05:42.425981 19316 ts_manager.cc:194] Re-registered known tserver with Master: ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101)
I20240503 08:05:42.426499 19316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48219
I20240503 08:05:42.608974 19576 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:42.609637 19442 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:42.610311 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.610378 20459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.622021 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:42.707168 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.708964 19576 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:42.709142 19442 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:42.718459 20459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.727983 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:42.831642 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:42.831920 19442 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:42.832237 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.833405 19576 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:42.833443 20459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.925981 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.926084 19576 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:42.926991 20459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:42.928893 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:42.931257 19442 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:42.975809 19464 consensus_queue.cc:237] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:42.977351 19595 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:42.977650 19728 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:42.977859 20673 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:42.978063 20184 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:42.980051 20672 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:42.980561 19595 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:42.980628 19728 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:42.983062 19319 catalog_manager.cc:5484] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:42.985937 19397 consensus_peers.cc:485] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad -> Peer 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637): Couldn't send request to peer 2d709df031144ec6b5c0fd04315bd1fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54c2f20ffc844500adc57b6961b3e9d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:42.990823 19595 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:42.992158 19464 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:42.992398 19728 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:42.992692 20687 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:42.992880 20688 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:42.994645 20687 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:42.995981 19464 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
W20240503 08:05:42.996825 19529 consensus_peers.cc:485] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 -> Peer ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101): Couldn't send request to peer ea59580caefd4780bb4f019a0133e704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc872d506be04591a178b33ca06d09f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:42.997304 19728 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:42.997340 19316 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:43.006177 19595 consensus_queue.cc:237] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.007449 19464 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.007647 19728 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.008002 20687 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.008260 20688 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.009529 20190 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.009941 19464 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.011909 19319 catalog_manager.cc:5484] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:43.012745 19529 consensus_peers.cc:485] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 -> Peer 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637): Couldn't send request to peer 2d709df031144ec6b5c0fd04315bd1fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 170ea8d950d4437eb5c3bdae4956741c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:43.012950 19728 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.019750 19464 consensus_queue.cc:237] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.021075 19728 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.021384 19595 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.021448 20673 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.021797 20184 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.023216 20184 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:43.023736 19728 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:43.024129 19595 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
W20240503 08:05:43.025784 19397 consensus_peers.cc:485] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad -> Peer ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101): Couldn't send request to peer ea59580caefd4780bb4f019a0133e704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08dc753bd09c48d4baa70f08b643cfec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:43.026295 19319 catalog_manager.cc:5484] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:43.033056 19595 consensus_queue.cc:237] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.034319 19728 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.034412 19464 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:05:43.034492 19529 consensus_peers.cc:485] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 -> Peer ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101): Couldn't send request to peer ea59580caefd4780bb4f019a0133e704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46e7a4621fd0462c9feeb0f454e44d03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:43.034675 20687 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.034863 20190 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.036412 20687 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:43.036870 19464 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:43.037124 19728 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:43.038790 19316 catalog_manager.cc:5484] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:43.045708 19728 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.047504 19464 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.048023 20692 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:43.049962 20692 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.050192 19464 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.051160 19595 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:43.051646 20251 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:05:43.052664 19661 consensus_peers.cc:485] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 -> Peer 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637): Couldn't send request to peer 2d709df031144ec6b5c0fd04315bd1fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40ea96d08bcb46028cc7f907b489e76b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:43.052634 19595 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.053769 19319 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:43.255631 20394 heartbeater.cc:503] Master 127.0.99.190:43309 was elected leader, sending a full tablet report...
I20240503 08:05:43.428333 20525 heartbeater.cc:503] Master 127.0.99.190:43309 was elected leader, sending a full tablet report...
I20240503 08:05:43.434561 20781 ts_tablet_manager.cc:902] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Initiating tablet copy from peer f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:43.435272 20781 tablet_copy_client.cc:308] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45203
I20240503 08:05:43.438704 19748 tablet_copy_service.cc:140] P f835c24114954b93a22708d01227b666: Received BeginTabletCopySession request for tablet 40ea96d08bcb46028cc7f907b489e76b from peer 2d709df031144ec6b5c0fd04315bd1fb ({username='slave'} at 127.0.99.133:44663)
I20240503 08:05:43.438849 19748 tablet_copy_service.cc:161] P f835c24114954b93a22708d01227b666: Beginning new tablet copy session on tablet 40ea96d08bcb46028cc7f907b489e76b from peer 2d709df031144ec6b5c0fd04315bd1fb at {username='slave'} at 127.0.99.133:44663: session id = 2d709df031144ec6b5c0fd04315bd1fb-40ea96d08bcb46028cc7f907b489e76b
I20240503 08:05:43.440014 19748 tablet_copy_source_session.cc:205] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.443176 20781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ea96d08bcb46028cc7f907b489e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.448307 20781 tablet_copy_client.cc:783] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.448473 20781 tablet_copy_client.cc:647] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.450084 20781 tablet_copy_client.cc:523] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.451722 20781 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap starting.
I20240503 08:05:43.452944 20787 ts_tablet_manager.cc:902] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Initiating tablet copy from peer 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199)
I20240503 08:05:43.453439 20787 tablet_copy_client.cc:308] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46199
I20240503 08:05:43.457773 19616 tablet_copy_service.cc:140] P 563200d2e6694145a13a4fd97d23b4f4: Received BeginTabletCopySession request for tablet 170ea8d950d4437eb5c3bdae4956741c from peer 2d709df031144ec6b5c0fd04315bd1fb ({username='slave'} at 127.0.99.133:42681)
I20240503 08:05:43.457908 19616 tablet_copy_service.cc:161] P 563200d2e6694145a13a4fd97d23b4f4: Beginning new tablet copy session on tablet 170ea8d950d4437eb5c3bdae4956741c from peer 2d709df031144ec6b5c0fd04315bd1fb at {username='slave'} at 127.0.99.133:42681: session id = 2d709df031144ec6b5c0fd04315bd1fb-170ea8d950d4437eb5c3bdae4956741c
I20240503 08:05:43.458596 20781 log.cc:824] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:43.459287 19616 tablet_copy_source_session.cc:205] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.460383 20787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170ea8d950d4437eb5c3bdae4956741c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.462421 20787 tablet_copy_client.cc:783] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.462579 20787 tablet_copy_client.cc:647] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.464093 20787 tablet_copy_client.cc:523] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.464323 20793 ts_tablet_manager.cc:902] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Initiating tablet copy from peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:43.464821 20793 tablet_copy_client.cc:308] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:44963
I20240503 08:05:43.465422 20787 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap starting.
I20240503 08:05:43.469108 19484 tablet_copy_service.cc:140] P b38441293bcc4b57b5f4c09adeaf70ad: Received BeginTabletCopySession request for tablet 54c2f20ffc844500adc57b6961b3e9d5 from peer 2d709df031144ec6b5c0fd04315bd1fb ({username='slave'} at 127.0.99.133:55339)
I20240503 08:05:43.469275 19484 tablet_copy_service.cc:161] P b38441293bcc4b57b5f4c09adeaf70ad: Beginning new tablet copy session on tablet 54c2f20ffc844500adc57b6961b3e9d5 from peer 2d709df031144ec6b5c0fd04315bd1fb at {username='slave'} at 127.0.99.133:55339: session id = 2d709df031144ec6b5c0fd04315bd1fb-54c2f20ffc844500adc57b6961b3e9d5
I20240503 08:05:43.470654 19484 tablet_copy_source_session.cc:205] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.471904 20793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c2f20ffc844500adc57b6961b3e9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.473863 20793 tablet_copy_client.cc:783] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.474005 20793 tablet_copy_client.cc:647] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.475409 20793 tablet_copy_client.cc:523] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.476485 20793 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap starting.
I20240503 08:05:43.483475 20787 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.483789 20787 tablet_bootstrap.cc:492] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap complete.
I20240503 08:05:43.484077 20787 ts_tablet_manager.cc:1367] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240503 08:05:43.484292 20781 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.484542 20781 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap complete.
I20240503 08:05:43.484687 20781 ts_tablet_manager.cc:1367] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.001s
I20240503 08:05:43.486261 20781 raft_consensus.cc:348] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.486261 20787 raft_consensus.cc:348] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.486620 20781 raft_consensus.cc:729] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d709df031144ec6b5c0fd04315bd1fb, State: Initialized, Role: LEARNER
I20240503 08:05:43.486620 20787 raft_consensus.cc:729] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d709df031144ec6b5c0fd04315bd1fb, State: Initialized, Role: LEARNER
I20240503 08:05:43.487011 20781 consensus_queue.cc:260] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.487011 20787 consensus_queue.cc:260] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.487879 20781 ts_tablet_manager.cc:1398] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 08:05:43.487880 20787 ts_tablet_manager.cc:1398] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:05:43.488653 19748 tablet_copy_service.cc:342] P f835c24114954b93a22708d01227b666: Request end of tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-40ea96d08bcb46028cc7f907b489e76b received from {username='slave'} at 127.0.99.133:44663
I20240503 08:05:43.488647 19616 tablet_copy_service.cc:342] P 563200d2e6694145a13a4fd97d23b4f4: Request end of tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-170ea8d950d4437eb5c3bdae4956741c received from {username='slave'} at 127.0.99.133:42681
I20240503 08:05:43.488749 19748 tablet_copy_service.cc:433] P f835c24114954b93a22708d01227b666: ending tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-40ea96d08bcb46028cc7f907b489e76b on tablet 40ea96d08bcb46028cc7f907b489e76b with peer 2d709df031144ec6b5c0fd04315bd1fb
I20240503 08:05:43.488750 19616 tablet_copy_service.cc:433] P 563200d2e6694145a13a4fd97d23b4f4: ending tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-170ea8d950d4437eb5c3bdae4956741c on tablet 170ea8d950d4437eb5c3bdae4956741c with peer 2d709df031144ec6b5c0fd04315bd1fb
I20240503 08:05:43.489482 20800 ts_tablet_manager.cc:902] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Initiating tablet copy from peer 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199)
I20240503 08:05:43.490502 20800 tablet_copy_client.cc:308] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46199
I20240503 08:05:43.494628 19616 tablet_copy_service.cc:140] P 563200d2e6694145a13a4fd97d23b4f4: Received BeginTabletCopySession request for tablet 46e7a4621fd0462c9feeb0f454e44d03 from peer ea59580caefd4780bb4f019a0133e704 ({username='slave'} at 127.0.99.132:34891)
I20240503 08:05:43.494735 19616 tablet_copy_service.cc:161] P 563200d2e6694145a13a4fd97d23b4f4: Beginning new tablet copy session on tablet 46e7a4621fd0462c9feeb0f454e44d03 from peer ea59580caefd4780bb4f019a0133e704 at {username='slave'} at 127.0.99.132:34891: session id = ea59580caefd4780bb4f019a0133e704-46e7a4621fd0462c9feeb0f454e44d03
I20240503 08:05:43.495625 19616 tablet_copy_source_session.cc:205] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.498220 20793 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.498428 20800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7a4621fd0462c9feeb0f454e44d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.498451 20793 tablet_bootstrap.cc:492] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap complete.
I20240503 08:05:43.498598 20793 ts_tablet_manager.cc:1367] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20240503 08:05:43.499089 20793 raft_consensus.cc:348] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.499207 20793 raft_consensus.cc:729] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d709df031144ec6b5c0fd04315bd1fb, State: Initialized, Role: LEARNER
I20240503 08:05:43.499305 20793 consensus_queue.cc:260] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:43.499567 20793 ts_tablet_manager.cc:1398] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:05:43.500136 19484 tablet_copy_service.cc:342] P b38441293bcc4b57b5f4c09adeaf70ad: Request end of tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-54c2f20ffc844500adc57b6961b3e9d5 received from {username='slave'} at 127.0.99.133:55339
I20240503 08:05:43.500221 19484 tablet_copy_service.cc:433] P b38441293bcc4b57b5f4c09adeaf70ad: ending tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-54c2f20ffc844500adc57b6961b3e9d5 on tablet 54c2f20ffc844500adc57b6961b3e9d5 with peer 2d709df031144ec6b5c0fd04315bd1fb
I20240503 08:05:43.504488 20800 tablet_copy_client.cc:783] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.504663 20800 tablet_copy_client.cc:647] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.505981 20800 tablet_copy_client.cc:523] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.507479 20800 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Bootstrap starting.
I20240503 08:05:43.513103 20800 log.cc:824] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:43.535634 20800 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.536028 20800 tablet_bootstrap.cc:492] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Bootstrap complete.
I20240503 08:05:43.536372 20800 ts_tablet_manager.cc:1367] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20240503 08:05:43.538854 20800 raft_consensus.cc:348] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.539124 20800 raft_consensus.cc:729] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea59580caefd4780bb4f019a0133e704, State: Initialized, Role: LEARNER
I20240503 08:05:43.539505 20800 consensus_queue.cc:260] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.540313 20830 ts_tablet_manager.cc:902] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Initiating tablet copy from peer 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199)
I20240503 08:05:43.540819 20830 tablet_copy_client.cc:308] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46199
I20240503 08:05:43.541261 19616 tablet_copy_service.cc:140] P 563200d2e6694145a13a4fd97d23b4f4: Received BeginTabletCopySession request for tablet bc872d506be04591a178b33ca06d09f2 from peer ea59580caefd4780bb4f019a0133e704 ({username='slave'} at 127.0.99.132:34891)
I20240503 08:05:43.541350 19616 tablet_copy_service.cc:161] P 563200d2e6694145a13a4fd97d23b4f4: Beginning new tablet copy session on tablet bc872d506be04591a178b33ca06d09f2 from peer ea59580caefd4780bb4f019a0133e704 at {username='slave'} at 127.0.99.132:34891: session id = ea59580caefd4780bb4f019a0133e704-bc872d506be04591a178b33ca06d09f2
I20240503 08:05:43.541357 20800 ts_tablet_manager.cc:1398] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:05:43.542230 19616 tablet_copy_source_session.cc:205] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.542666 19616 tablet_copy_service.cc:342] P 563200d2e6694145a13a4fd97d23b4f4: Request end of tablet copy session ea59580caefd4780bb4f019a0133e704-46e7a4621fd0462c9feeb0f454e44d03 received from {username='slave'} at 127.0.99.132:34891
I20240503 08:05:43.542748 19616 tablet_copy_service.cc:433] P 563200d2e6694145a13a4fd97d23b4f4: ending tablet copy session ea59580caefd4780bb4f019a0133e704-46e7a4621fd0462c9feeb0f454e44d03 on tablet 46e7a4621fd0462c9feeb0f454e44d03 with peer ea59580caefd4780bb4f019a0133e704
I20240503 08:05:43.543440 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc872d506be04591a178b33ca06d09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.545615 20830 tablet_copy_client.cc:783] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.545773 20830 tablet_copy_client.cc:647] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.546857 20830 tablet_copy_client.cc:523] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.548059 20830 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Bootstrap starting.
I20240503 08:05:43.565824 20830 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.566077 20830 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Bootstrap complete.
I20240503 08:05:43.566232 20830 ts_tablet_manager.cc:1367] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 08:05:43.566718 20830 raft_consensus.cc:348] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.566828 20830 raft_consensus.cc:729] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea59580caefd4780bb4f019a0133e704, State: Initialized, Role: LEARNER
I20240503 08:05:43.566942 20830 consensus_queue.cc:260] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.567175 20830 ts_tablet_manager.cc:1398] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:05:43.567713 19616 tablet_copy_service.cc:342] P 563200d2e6694145a13a4fd97d23b4f4: Request end of tablet copy session ea59580caefd4780bb4f019a0133e704-bc872d506be04591a178b33ca06d09f2 received from {username='slave'} at 127.0.99.132:34891
I20240503 08:05:43.567797 19616 tablet_copy_service.cc:433] P 563200d2e6694145a13a4fd97d23b4f4: ending tablet copy session ea59580caefd4780bb4f019a0133e704-bc872d506be04591a178b33ca06d09f2 on tablet bc872d506be04591a178b33ca06d09f2 with peer ea59580caefd4780bb4f019a0133e704
I20240503 08:05:43.590246 20830 ts_tablet_manager.cc:902] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Initiating tablet copy from peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:43.590654 20830 tablet_copy_client.cc:308] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:44963
I20240503 08:05:43.593624 19484 tablet_copy_service.cc:140] P b38441293bcc4b57b5f4c09adeaf70ad: Received BeginTabletCopySession request for tablet 08dc753bd09c48d4baa70f08b643cfec from peer ea59580caefd4780bb4f019a0133e704 ({username='slave'} at 127.0.99.132:42105)
I20240503 08:05:43.593732 19484 tablet_copy_service.cc:161] P b38441293bcc4b57b5f4c09adeaf70ad: Beginning new tablet copy session on tablet 08dc753bd09c48d4baa70f08b643cfec from peer ea59580caefd4780bb4f019a0133e704 at {username='slave'} at 127.0.99.132:42105: session id = ea59580caefd4780bb4f019a0133e704-08dc753bd09c48d4baa70f08b643cfec
I20240503 08:05:43.594434 19484 tablet_copy_source_session.cc:205] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:43.595206 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08dc753bd09c48d4baa70f08b643cfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:43.596838 20830 tablet_copy_client.cc:783] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:43.596974 20830 tablet_copy_client.cc:647] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:43.597858 20830 tablet_copy_client.cc:523] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:43.598753 20830 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Bootstrap starting.
I20240503 08:05:43.616611 20830 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:43.616866 20830 tablet_bootstrap.cc:492] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Bootstrap complete.
I20240503 08:05:43.617028 20830 ts_tablet_manager.cc:1367] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 08:05:43.617400 20830 raft_consensus.cc:348] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.617504 20830 raft_consensus.cc:729] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea59580caefd4780bb4f019a0133e704, State: Initialized, Role: LEARNER
I20240503 08:05:43.617599 20830 consensus_queue.cc:260] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:43.617820 20830 ts_tablet_manager.cc:1398] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:43.618396 19484 tablet_copy_service.cc:342] P b38441293bcc4b57b5f4c09adeaf70ad: Request end of tablet copy session ea59580caefd4780bb4f019a0133e704-08dc753bd09c48d4baa70f08b643cfec received from {username='slave'} at 127.0.99.132:42105
I20240503 08:05:43.618477 19484 tablet_copy_service.cc:433] P b38441293bcc4b57b5f4c09adeaf70ad: ending tablet copy session ea59580caefd4780bb4f019a0133e704-08dc753bd09c48d4baa70f08b643cfec on tablet 08dc753bd09c48d4baa70f08b643cfec with peer ea59580caefd4780bb4f019a0133e704
I20240503 08:05:43.837154 20348 raft_consensus.cc:1203] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:43.868129 20348 raft_consensus.cc:1203] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:43.948572 20479 raft_consensus.cc:1203] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:44.009496 20348 raft_consensus.cc:1203] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:44.025709 20479 raft_consensus.cc:1203] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:44.030366 20479 raft_consensus.cc:1203] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:44.287969 20884 raft_consensus.cc:1050] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad: attempting to promote NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb to VOTER
I20240503 08:05:44.288434 20884 consensus_queue.cc:237] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.289793 19595 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.289808 20348 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.289839 19728 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.290199 20947 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.290402 20884 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.290560 20673 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.292356 20884 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.292760 19728 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.294466 19595 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.294420 20348 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.295140 19321 catalog_manager.cc:5484] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 reported cstate change: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.299539 19463 consensus_queue.cc:237] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.300518 20348 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.300551 19728 raft_consensus.cc:1261] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.300869 20673 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.301194 20673 consensus_queue.cc:1035] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.302573 20885 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.302894 20348 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.303994 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54c2f20ffc844500adc57b6961b3e9d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.304358 19728 raft_consensus.cc:2932] T 54c2f20ffc844500adc57b6961b3e9d5 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.304682 19316 catalog_manager.cc:5484] T 54c2f20ffc844500adc57b6961b3e9d5 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:44.307049 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c2f20ffc844500adc57b6961b3e9d5 with delete_type TABLET_DATA_TOMBSTONED (TS 563200d2e6694145a13a4fd97d23b4f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38212
I20240503 08:05:44.307498 20955 tablet_replica.cc:330] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: stopping tablet replica
I20240503 08:05:44.307811 20955 raft_consensus.cc:2229] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.308033 20955 raft_consensus.cc:2258] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.309067 20955 ts_tablet_manager.cc:1875] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.310843 20955 ts_tablet_manager.cc:1888] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.310945 20955 log.cc:1196] T 54c2f20ffc844500adc57b6961b3e9d5 P 563200d2e6694145a13a4fd97d23b4f4: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/54c2f20ffc844500adc57b6961b3e9d5
I20240503 08:05:44.311388 19307 catalog_manager.cc:4830] TS 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199): tablet 54c2f20ffc844500adc57b6961b3e9d5 (table rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]) successfully deleted
I20240503 08:05:44.393743 20851 raft_consensus.cc:1050] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: attempting to promote NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb to VOTER
I20240503 08:05:44.394227 20851 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.395552 20348 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.395574 19463 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.395574 19595 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.395956 20957 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.396158 20692 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.396339 20958 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.398401 20958 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.398839 19463 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.398867 20348 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.401144 19319 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.401257 19595 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.404134 19728 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.405165 20348 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.405184 19463 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.405468 20958 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.405643 20957 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.407060 20851 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.407451 19463 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.408392 19306 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40ea96d08bcb46028cc7f907b489e76b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.408949 20348 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.409099 19316 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 reported cstate change: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:44.411410 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ea96d08bcb46028cc7f907b489e76b with delete_type TABLET_DATA_TOMBSTONED (TS 563200d2e6694145a13a4fd97d23b4f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38212
I20240503 08:05:44.411880 20955 tablet_replica.cc:330] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: stopping tablet replica
I20240503 08:05:44.412015 20955 raft_consensus.cc:2229] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.412122 20955 raft_consensus.cc:2258] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.412864 20955 ts_tablet_manager.cc:1875] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.414773 20955 ts_tablet_manager.cc:1888] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.414850 20955 log.cc:1196] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/40ea96d08bcb46028cc7f907b489e76b
I20240503 08:05:44.415361 19307 catalog_manager.cc:4830] TS 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199): tablet 40ea96d08bcb46028cc7f907b489e76b (table rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]) successfully deleted
I20240503 08:05:44.450367 20947 raft_consensus.cc:1050] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad: attempting to promote NON_VOTER ea59580caefd4780bb4f019a0133e704 to VOTER
I20240503 08:05:44.450748 20947 consensus_queue.cc:237] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.452153 19595 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.452239 20479 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.452559 20673 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.452657 19728 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.453024 20884 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.453317 20884 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.455166 20884 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.455660 19728 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.457470 19595 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.457950 19321 catalog_manager.cc:5484] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:44.459903 20479 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.461494 20883 raft_consensus.cc:1050] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4: attempting to promote NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb to VOTER
I20240503 08:05:44.462438 20883 consensus_queue.cc:237] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.463902 19463 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.464848 20348 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.464848 19728 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.465234 20972 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.465770 19463 consensus_queue.cc:237] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.465816 20972 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.465955 20688 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.467041 20479 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.467429 19728 raft_consensus.cc:1261] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.467968 20884 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.468186 20673 consensus_queue.cc:1035] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.468284 20688 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.468859 19464 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.469967 20947 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.470407 19728 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.471166 20348 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.471516 19321 catalog_manager.cc:5484] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:44.471966 20479 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.472606 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08dc753bd09c48d4baa70f08b643cfec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.473805 19728 raft_consensus.cc:2932] T 08dc753bd09c48d4baa70f08b643cfec P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.474820 19321 catalog_manager.cc:5484] T 08dc753bd09c48d4baa70f08b643cfec P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 5 to 6, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.476540 19595 consensus_queue.cc:237] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:44.477324 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet 08dc753bd09c48d4baa70f08b643cfec with delete_type TABLET_DATA_TOMBSTONED (TS 563200d2e6694145a13a4fd97d23b4f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38212
I20240503 08:05:44.477502 20955 tablet_replica.cc:330] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: stopping tablet replica
I20240503 08:05:44.477622 20955 raft_consensus.cc:2229] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.477629 19464 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.477757 20955 raft_consensus.cc:2258] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.478241 20882 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.478302 20348 raft_consensus.cc:1261] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.478336 20955 ts_tablet_manager.cc:1875] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.478633 20972 consensus_queue.cc:1035] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.480072 20883 raft_consensus.cc:1050] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4: attempting to promote NON_VOTER ea59580caefd4780bb4f019a0133e704 to VOTER
I20240503 08:05:44.480183 20972 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.480296 20955 ts_tablet_manager.cc:1888] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.480357 20955 log.cc:1196] T 08dc753bd09c48d4baa70f08b643cfec P 563200d2e6694145a13a4fd97d23b4f4: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/08dc753bd09c48d4baa70f08b643cfec
I20240503 08:05:44.480394 20883 consensus_queue.cc:237] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.481750 19463 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.481658 19464 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.482157 20882 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.482313 19319 catalog_manager.cc:5484] T 170ea8d950d4437eb5c3bdae4956741c P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:44.482352 20348 raft_consensus.cc:2932] T 170ea8d950d4437eb5c3bdae4956741c P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:44.483067 19307 catalog_manager.cc:4830] TS 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199): tablet 08dc753bd09c48d4baa70f08b643cfec (table rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]) successfully deleted
I20240503 08:05:44.483151 19728 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.483220 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 170ea8d950d4437eb5c3bdae4956741c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.483901 20882 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.484055 20479 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.484458 20984 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.486361 20981 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.486783 19707 tablet_service.cc:1514] Processing DeleteTablet for tablet 170ea8d950d4437eb5c3bdae4956741c with delete_type TABLET_DATA_TOMBSTONED (TS f835c24114954b93a22708d01227b666 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55674
I20240503 08:05:44.487828 20479 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.488008 19463 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.488234 20988 tablet_replica.cc:330] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: stopping tablet replica
I20240503 08:05:44.488543 20988 raft_consensus.cc:2229] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.488718 20988 raft_consensus.cc:2258] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.489763 19321 catalog_manager.cc:5484] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.491509 20988 ts_tablet_manager.cc:1875] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.492545 19728 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.493402 19595 consensus_queue.cc:237] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.494518 20988 ts_tablet_manager.cc:1888] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.494606 20988 log.cc:1196] T 170ea8d950d4437eb5c3bdae4956741c P f835c24114954b93a22708d01227b666: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/170ea8d950d4437eb5c3bdae4956741c
I20240503 08:05:44.495082 20479 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.495319 19306 catalog_manager.cc:4830] TS f835c24114954b93a22708d01227b666 (127.0.99.131:45203): tablet 170ea8d950d4437eb5c3bdae4956741c (table rebalance_test_table_2 [id=1a64d3b4a80a44a5b82a4580d3703d74]) successfully deleted
I20240503 08:05:44.495409 20882 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.495784 19464 raft_consensus.cc:1261] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.496122 20882 consensus_queue.cc:1035] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.497431 20688 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.497925 20479 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.498723 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46e7a4621fd0462c9feeb0f454e44d03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.499007 19464 raft_consensus.cc:2932] T 46e7a4621fd0462c9feeb0f454e44d03 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.500332 19320 catalog_manager.cc:5484] T 46e7a4621fd0462c9feeb0f454e44d03 P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 5 to 6, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.502535 19707 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7a4621fd0462c9feeb0f454e44d03 with delete_type TABLET_DATA_TOMBSTONED (TS f835c24114954b93a22708d01227b666 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55674
I20240503 08:05:44.502737 20988 tablet_replica.cc:330] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: stopping tablet replica
I20240503 08:05:44.502854 20988 raft_consensus.cc:2229] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.503011 20988 raft_consensus.cc:2258] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.503638 20988 ts_tablet_manager.cc:1875] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.505416 20988 ts_tablet_manager.cc:1888] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.505486 20988 log.cc:1196] T 46e7a4621fd0462c9feeb0f454e44d03 P f835c24114954b93a22708d01227b666: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/46e7a4621fd0462c9feeb0f454e44d03
I20240503 08:05:44.505959 19306 catalog_manager.cc:4830] TS f835c24114954b93a22708d01227b666 (127.0.99.131:45203): tablet 46e7a4621fd0462c9feeb0f454e44d03 (table rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]) successfully deleted
I20240503 08:05:44.603825 20882 raft_consensus.cc:1050] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: attempting to promote NON_VOTER ea59580caefd4780bb4f019a0133e704 to VOTER
I20240503 08:05:44.604211 20882 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.605509 20479 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.605535 19464 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.605576 19728 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:44.605928 20882 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.606132 20983 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.606316 20972 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:44.608088 20972 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.608456 19464 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.608456 19728 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.610517 20479 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.610869 19316 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:44.613843 19595 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:44.615468 19728 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.615816 20972 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.616032 20479 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:44.616317 20972 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:44.617197 20983 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.617628 19728 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.618078 20479 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.618553 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc872d506be04591a178b33ca06d09f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:44.619642 19320 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 reported cstate change: config changed from index 5 to 6, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:44.621788 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet bc872d506be04591a178b33ca06d09f2 with delete_type TABLET_DATA_TOMBSTONED (TS b38441293bcc4b57b5f4c09adeaf70ad not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37106
I20240503 08:05:44.622340 21018 tablet_replica.cc:330] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: stopping tablet replica
I20240503 08:05:44.622609 21018 raft_consensus.cc:2229] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:44.622807 21018 raft_consensus.cc:2258] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:44.623765 21018 ts_tablet_manager.cc:1875] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:44.625427 21018 ts_tablet_manager.cc:1888] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:44.625499 21018 log.cc:1196] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bc872d506be04591a178b33ca06d09f2
I20240503 08:05:44.625977 19307 catalog_manager.cc:4830] TS b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): tablet bc872d506be04591a178b33ca06d09f2 (table rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]) successfully deleted
I20240503 08:05:44.775247 19576 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:05:44.778100 20459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:44.778334 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:44.779141 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:44.783219 19442 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:44.816588 19464 consensus_queue.cc:237] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:44.817956 19595 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:44.818079 19728 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:05:44.818099 19397 consensus_peers.cc:485] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad -> Peer 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637): Couldn't send request to peer 2d709df031144ec6b5c0fd04315bd1fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8530aaa9e8a43e58070765927a75798. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:44.818353 20673 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:44.818539 20947 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:44.820202 20884 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:44.820586 19595 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:44.820629 19728 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } } }
I20240503 08:05:44.822757 19319 catalog_manager.cc:5484] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index -1 to 4, NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:44.830298 19464 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:44.831615 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:44.831933 19728 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:44.831974 20885 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:44.832320 20885 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:05:44.832548 19397 consensus_peers.cc:485] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad -> Peer ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101): Couldn't send request to peer ea59580caefd4780bb4f019a0133e704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb2dde0bfa18445d8b4a05db1030ef65. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:44.833937 20673 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:44.834252 19595 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:44.834326 19728 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:44.836709 19321 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index -1 to 4, NON_VOTER ea59580caefd4780bb4f019a0133e704 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } } }
I20240503 08:05:45.229485 21090 ts_tablet_manager.cc:902] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Initiating tablet copy from peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:45.230032 21090 tablet_copy_client.cc:308] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:44963
I20240503 08:05:45.230507 19484 tablet_copy_service.cc:140] P b38441293bcc4b57b5f4c09adeaf70ad: Received BeginTabletCopySession request for tablet eb2dde0bfa18445d8b4a05db1030ef65 from peer ea59580caefd4780bb4f019a0133e704 ({username='slave'} at 127.0.99.132:42105)
I20240503 08:05:45.230604 19484 tablet_copy_service.cc:161] P b38441293bcc4b57b5f4c09adeaf70ad: Beginning new tablet copy session on tablet eb2dde0bfa18445d8b4a05db1030ef65 from peer ea59580caefd4780bb4f019a0133e704 at {username='slave'} at 127.0.99.132:42105: session id = ea59580caefd4780bb4f019a0133e704-eb2dde0bfa18445d8b4a05db1030ef65
I20240503 08:05:45.231400 19484 tablet_copy_source_session.cc:205] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:45.232199 21090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2dde0bfa18445d8b4a05db1030ef65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:45.233843 21090 tablet_copy_client.cc:783] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:45.233983 21090 tablet_copy_client.cc:647] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:45.235036 21090 tablet_copy_client.cc:523] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:45.235947 21090 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Bootstrap starting.
I20240503 08:05:45.252239 21090 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:45.252504 21090 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Bootstrap complete.
I20240503 08:05:45.252671 21090 ts_tablet_manager.cc:1367] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240503 08:05:45.253065 21090 raft_consensus.cc:348] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:45.253175 21090 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea59580caefd4780bb4f019a0133e704, State: Initialized, Role: LEARNER
I20240503 08:05:45.253268 21090 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: true } }
I20240503 08:05:45.253594 21090 ts_tablet_manager.cc:1398] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:05:45.254113 19484 tablet_copy_service.cc:342] P b38441293bcc4b57b5f4c09adeaf70ad: Request end of tablet copy session ea59580caefd4780bb4f019a0133e704-eb2dde0bfa18445d8b4a05db1030ef65 received from {username='slave'} at 127.0.99.132:42105
I20240503 08:05:45.254189 19484 tablet_copy_service.cc:433] P b38441293bcc4b57b5f4c09adeaf70ad: ending tablet copy session ea59580caefd4780bb4f019a0133e704-eb2dde0bfa18445d8b4a05db1030ef65 on tablet eb2dde0bfa18445d8b4a05db1030ef65 with peer ea59580caefd4780bb4f019a0133e704
I20240503 08:05:45.359998 21107 ts_tablet_manager.cc:902] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Initiating tablet copy from peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963)
I20240503 08:05:45.360442 21107 tablet_copy_client.cc:308] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:44963
I20240503 08:05:45.360911 19484 tablet_copy_service.cc:140] P b38441293bcc4b57b5f4c09adeaf70ad: Received BeginTabletCopySession request for tablet f8530aaa9e8a43e58070765927a75798 from peer 2d709df031144ec6b5c0fd04315bd1fb ({username='slave'} at 127.0.99.133:55339)
I20240503 08:05:45.361017 19484 tablet_copy_service.cc:161] P b38441293bcc4b57b5f4c09adeaf70ad: Beginning new tablet copy session on tablet f8530aaa9e8a43e58070765927a75798 from peer 2d709df031144ec6b5c0fd04315bd1fb at {username='slave'} at 127.0.99.133:55339: session id = 2d709df031144ec6b5c0fd04315bd1fb-f8530aaa9e8a43e58070765927a75798
I20240503 08:05:45.361727 19484 tablet_copy_source_session.cc:205] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:45.362479 21107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8530aaa9e8a43e58070765927a75798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:45.364163 21107 tablet_copy_client.cc:783] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:45.364303 21107 tablet_copy_client.cc:647] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:45.365337 21107 tablet_copy_client.cc:523] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:45.366258 21107 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap starting.
I20240503 08:05:45.382867 21107 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:45.383241 21107 tablet_bootstrap.cc:492] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Bootstrap complete.
I20240503 08:05:45.383419 21107 ts_tablet_manager.cc:1367] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.003s
I20240503 08:05:45.383816 21107 raft_consensus.cc:348] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:45.383924 21107 raft_consensus.cc:729] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d709df031144ec6b5c0fd04315bd1fb, State: Initialized, Role: LEARNER
I20240503 08:05:45.384016 21107 consensus_queue.cc:260] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: true } }
I20240503 08:05:45.384411 21107 ts_tablet_manager.cc:1398] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:45.384991 19484 tablet_copy_service.cc:342] P b38441293bcc4b57b5f4c09adeaf70ad: Request end of tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-f8530aaa9e8a43e58070765927a75798 received from {username='slave'} at 127.0.99.133:55339
I20240503 08:05:45.385082 19484 tablet_copy_service.cc:433] P b38441293bcc4b57b5f4c09adeaf70ad: ending tablet copy session 2d709df031144ec6b5c0fd04315bd1fb-f8530aaa9e8a43e58070765927a75798 on tablet f8530aaa9e8a43e58070765927a75798 with peer 2d709df031144ec6b5c0fd04315bd1fb
I20240503 08:05:45.628389 20479 raft_consensus.cc:1203] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:45.896154 20348 raft_consensus.cc:1203] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:46.121963 20885 raft_consensus.cc:1050] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: attempting to promote NON_VOTER ea59580caefd4780bb4f019a0133e704 to VOTER
I20240503 08:05:46.122309 20885 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:46.123485 20479 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 LEARNER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.123550 19728 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.123561 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.123929 21125 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.124133 21160 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.124292 21124 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.125978 21160 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:46.126425 19595 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:46.127068 19728 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:46.129163 19319 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:46.129613 20479 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea59580caefd4780bb4f019a0133e704 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:46.330147 19463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65"
dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59648
I20240503 08:05:46.330312 19463 raft_consensus.cc:595] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Received request to transfer leadership
I20240503 08:05:46.488919 20885 raft_consensus.cc:1050] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: attempting to promote NON_VOTER 2d709df031144ec6b5c0fd04315bd1fb to VOTER
I20240503 08:05:46.489297 20885 consensus_queue.cc:237] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:46.490576 20348 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 LEARNER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.490577 19595 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.490588 19728 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer b38441293bcc4b57b5f4c09adeaf70ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:46.490968 21160 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.491154 21125 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.491318 20885 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:46.493067 20885 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:46.493434 19595 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:46.493945 19728 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:46.494853 20348 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:46.496112 19321 catalog_manager.cc:5484] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 4 to 5, 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:46.545426 19463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65"
dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59670
I20240503 08:05:46.545550 19463 raft_consensus.cc:595] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Received request to transfer leadership
I20240503 08:05:46.554740 19463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798"
dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59692
I20240503 08:05:46.554863 19463 raft_consensus.cc:595] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Received request to transfer leadership
I20240503 08:05:46.568845 21124 raft_consensus.cc:979] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: : Instructing follower ea59580caefd4780bb4f019a0133e704 to start an election
I20240503 08:05:46.568953 21124 raft_consensus.cc:1067] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Signalling peer ea59580caefd4780bb4f019a0133e704 to start an election
I20240503 08:05:46.569540 20479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65"
dest_uuid: "ea59580caefd4780bb4f019a0133e704"
from {username='slave'} at 127.0.99.129:43915
I20240503 08:05:46.569722 20479 raft_consensus.cc:482] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:05:46.569829 20479 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.570690 20479 raft_consensus.cc:504] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:46.571692 20479 leader_election.cc:288] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [CANDIDATE]: Term 2 election: Requested vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199), f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:46.571928 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "ea59580caefd4780bb4f019a0133e704" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:46.572042 19463 raft_consensus.cc:3032] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:05:46.572081 19463 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:46.572067 19595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "ea59580caefd4780bb4f019a0133e704" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
I20240503 08:05:46.572188 19595 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.572227 19463 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:46.572404 19463 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.572945 19595 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea59580caefd4780bb4f019a0133e704 in term 2.
I20240503 08:05:46.573321 19463 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea59580caefd4780bb4f019a0133e704 in term 2.
I20240503 08:05:46.573592 20413 leader_election.cc:302] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad, ea59580caefd4780bb4f019a0133e704; no voters:
I20240503 08:05:46.573787 21165 raft_consensus.cc:2781] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:05:46.573925 21165 raft_consensus.cc:686] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 LEADER]: Becoming Leader. State: Replica: ea59580caefd4780bb4f019a0133e704, State: Running, Role: LEADER
I20240503 08:05:46.574194 21165 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:46.575851 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2dde0bfa18445d8b4a05db1030ef65" candidate_uuid: "ea59580caefd4780bb4f019a0133e704" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:46.575973 19728 raft_consensus.cc:3037] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.576375 19319 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 reported cstate change: term changed from 1 to 2, leader changed from b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) to ea59580caefd4780bb4f019a0133e704 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "ea59580caefd4780bb4f019a0133e704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:46.576890 19728 raft_consensus.cc:2454] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea59580caefd4780bb4f019a0133e704 in term 2.
I20240503 08:05:46.769948 19463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798"
dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59712
I20240503 08:05:46.770077 19463 raft_consensus.cc:595] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Received request to transfer leadership
I20240503 08:05:46.914448 21125 raft_consensus.cc:979] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: : Instructing follower 563200d2e6694145a13a4fd97d23b4f4 to start an election
I20240503 08:05:46.914553 21125 raft_consensus.cc:1067] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Signalling peer 563200d2e6694145a13a4fd97d23b4f4 to start an election
I20240503 08:05:46.915040 19595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798"
dest_uuid: "563200d2e6694145a13a4fd97d23b4f4"
from {username='slave'} at 127.0.99.129:43849
I20240503 08:05:46.915213 19595 raft_consensus.cc:482] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:05:46.915287 19595 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.916100 19595 raft_consensus.cc:504] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:46.916514 19595 leader_election.cc:288] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 2 election: Requested vote from peers b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963), f835c24114954b93a22708d01227b666 (127.0.99.131:45203), 2d709df031144ec6b5c0fd04315bd1fb (127.0.99.133:44637)
I20240503 08:05:46.916783 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b38441293bcc4b57b5f4c09adeaf70ad"
I20240503 08:05:46.916903 19463 raft_consensus.cc:3032] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:05:46.916940 19463 raft_consensus.cc:729] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Running, Role: LEADER
I20240503 08:05:46.916904 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f835c24114954b93a22708d01227b666"
I20240503 08:05:46.917023 19726 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.917042 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8530aaa9e8a43e58070765927a75798" candidate_uuid: "563200d2e6694145a13a4fd97d23b4f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2d709df031144ec6b5c0fd04315bd1fb"
I20240503 08:05:46.917066 19463 consensus_queue.cc:260] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:46.917207 20348 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.917241 19463 raft_consensus.cc:3037] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:05:46.917937 19463 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 2.
I20240503 08:05:46.918000 19726 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 2.
I20240503 08:05:46.918133 20348 raft_consensus.cc:2454] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563200d2e6694145a13a4fd97d23b4f4 in term 2.
I20240503 08:05:46.918442 19530 leader_election.cc:302] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 563200d2e6694145a13a4fd97d23b4f4, b38441293bcc4b57b5f4c09adeaf70ad, f835c24114954b93a22708d01227b666; no voters:
I20240503 08:05:46.918614 20983 raft_consensus.cc:2781] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:05:46.918771 20983 raft_consensus.cc:686] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 LEADER]: Becoming Leader. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Running, Role: LEADER
I20240503 08:05:46.918910 20983 consensus_queue.cc:237] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:46.920501 19319 catalog_manager.cc:5484] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: term changed from 1 to 2, leader changed from b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) to 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:46.989882 19463 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:46.990322 21165 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.035380 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:47.035900 21165 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.038877 19726 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:47.039343 21213 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.384874 19726 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:47.385365 20983 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.406006 19463 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:47.406505 20983 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.408671 20348 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 2 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:05:47.409807 21268 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:47.498817 20479 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } }
I20240503 08:05:47.499969 19726 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:05:47.499979 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:05:47.500377 21165 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:47.500584 21243 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:47.502096 21213 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:47.502491 19595 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:47.502565 19726 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } }
I20240503 08:05:47.503901 19306 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb2dde0bfa18445d8b4a05db1030ef65 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:47.504526 19320 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ea59580caefd4780bb4f019a0133e704" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:47.507094 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2dde0bfa18445d8b4a05db1030ef65 with delete_type TABLET_DATA_TOMBSTONED (TS b38441293bcc4b57b5f4c09adeaf70ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37106
I20240503 08:05:47.507431 21276 tablet_replica.cc:330] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: stopping tablet replica
I20240503 08:05:47.507561 21276 raft_consensus.cc:2229] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:47.507684 21276 raft_consensus.cc:2258] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:47.508244 21276 ts_tablet_manager.cc:1875] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:47.509896 21276 ts_tablet_manager.cc:1888] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:05:47.509958 21276 log.cc:1196] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/eb2dde0bfa18445d8b4a05db1030ef65
I20240503 08:05:47.510370 19307 catalog_manager.cc:4830] TS b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): tablet eb2dde0bfa18445d8b4a05db1030ef65 (table rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]) successfully deleted
I20240503 08:05:47.846855 19595 consensus_queue.cc:237] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } }
I20240503 08:05:47.848057 20347 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 2 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:05:47.848104 19726 raft_consensus.cc:1261] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:05:47.848439 20983 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:47.848637 21230 consensus_queue.cc:1035] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:47.850158 21268 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:47.850574 19726 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:47.850594 20347 raft_consensus.cc:2932] T f8530aaa9e8a43e58070765927a75798 P 2d709df031144ec6b5c0fd04315bd1fb [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } }
I20240503 08:05:47.851806 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8530aaa9e8a43e58070765927a75798 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:47.852630 19316 catalog_manager.cc:5484] T f8530aaa9e8a43e58070765927a75798 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 5 to 7, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:47.854749 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet f8530aaa9e8a43e58070765927a75798 with delete_type TABLET_DATA_TOMBSTONED (TS b38441293bcc4b57b5f4c09adeaf70ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37106
I20240503 08:05:47.855012 21276 tablet_replica.cc:330] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: stopping tablet replica
I20240503 08:05:47.855152 21276 raft_consensus.cc:2229] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:47.855258 21276 raft_consensus.cc:2258] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:47.855953 21276 ts_tablet_manager.cc:1875] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:47.857460 21276 ts_tablet_manager.cc:1888] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:05:47.857522 21276 log.cc:1196] T f8530aaa9e8a43e58070765927a75798 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8530aaa9e8a43e58070765927a75798
I20240503 08:05:47.857904 19307 catalog_manager.cc:4830] TS b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): tablet f8530aaa9e8a43e58070765927a75798 (table rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]) successfully deleted
I20240503 08:05:48.082409 20459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:48.082715 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:48.085765 19576 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:48.087567 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:48.098520 19442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:48.170523 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:48.170675 20459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:48.172125 19442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:48.177789 19576 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:48.182420 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:48.213968 19595 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.215377 20479 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:48.215680 19726 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:48.215811 20983 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:48.216118 21230 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:05:48.216279 19530 consensus_peers.cc:485] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 -> Peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): Couldn't send request to peer b38441293bcc4b57b5f4c09adeaf70ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:48.217542 20983 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.218082 20479 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.218348 19726 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.220575 19316 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 6 to 7, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.227622 20479 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.228838 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:05:48.228852 19726 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 08:05:48.229080 20413 consensus_peers.cc:485] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 -> Peer b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): Couldn't send request to peer b38441293bcc4b57b5f4c09adeaf70ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:48.229274 21387 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:48.229616 21386 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:48.230837 21387 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.231362 19726 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.232957 19595 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } } }
I20240503 08:05:48.233621 19320 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 reported cstate change: config changed from index 7 to 8, NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "ea59580caefd4780bb4f019a0133e704" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:48.241515 19726 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } }
I20240503 08:05:48.243065 20347 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:48.243108 19463 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:48.243544 21395 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:05:48.243758 19662 consensus_peers.cc:485] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 -> Peer 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199): Couldn't send request to peer 563200d2e6694145a13a4fd97d23b4f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:48.243952 21394 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:48.245659 21395 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } } }
I20240503 08:05:48.246191 20347 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } } }
I20240503 08:05:48.247735 19463 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } } }
I20240503 08:05:48.248186 19316 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb reported cstate change: config changed from index 6 to 7, NON_VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } } }
I20240503 08:05:48.696094 21438 ts_tablet_manager.cc:902] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Initiating tablet copy from peer ea59580caefd4780bb4f019a0133e704 (127.0.99.132:37101)
I20240503 08:05:48.696678 21438 tablet_copy_client.cc:308] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37101
I20240503 08:05:48.697330 20499 tablet_copy_service.cc:140] P ea59580caefd4780bb4f019a0133e704: Received BeginTabletCopySession request for tablet eb2dde0bfa18445d8b4a05db1030ef65 from peer b38441293bcc4b57b5f4c09adeaf70ad ({username='slave'} at 127.0.99.129:43915)
I20240503 08:05:48.697455 20499 tablet_copy_service.cc:161] P ea59580caefd4780bb4f019a0133e704: Beginning new tablet copy session on tablet eb2dde0bfa18445d8b4a05db1030ef65 from peer b38441293bcc4b57b5f4c09adeaf70ad at {username='slave'} at 127.0.99.129:43915: session id = b38441293bcc4b57b5f4c09adeaf70ad-eb2dde0bfa18445d8b4a05db1030ef65
I20240503 08:05:48.698729 20499 tablet_copy_source_session.cc:205] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:48.699934 21438 ts_tablet_manager.cc:1875] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:48.701939 21438 ts_tablet_manager.cc:1888] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 08:05:48.702095 21438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2dde0bfa18445d8b4a05db1030ef65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:48.703023 21438 tablet_copy_client.cc:783] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:48.703166 21438 tablet_copy_client.cc:647] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:48.704584 21438 tablet_copy_client.cc:523] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:48.705775 21438 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:48.723568 21438 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:48.723858 21438 tablet_bootstrap.cc:492] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap complete.
I20240503 08:05:48.724023 21438 ts_tablet_manager.cc:1367] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 08:05:48.724443 21438 raft_consensus.cc:348] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.724553 21438 raft_consensus.cc:729] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: LEARNER
I20240503 08:05:48.724646 21438 consensus_queue.cc:260] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.724915 21438 ts_tablet_manager.cc:1398] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:48.725607 20499 tablet_copy_service.cc:342] P ea59580caefd4780bb4f019a0133e704: Request end of tablet copy session b38441293bcc4b57b5f4c09adeaf70ad-eb2dde0bfa18445d8b4a05db1030ef65 received from {username='slave'} at 127.0.99.129:43915
I20240503 08:05:48.725695 20499 tablet_copy_service.cc:433] P ea59580caefd4780bb4f019a0133e704: ending tablet copy session b38441293bcc4b57b5f4c09adeaf70ad-eb2dde0bfa18445d8b4a05db1030ef65 on tablet eb2dde0bfa18445d8b4a05db1030ef65 with peer b38441293bcc4b57b5f4c09adeaf70ad
I20240503 08:05:48.794688 21438 ts_tablet_manager.cc:902] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Initiating tablet copy from peer 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199)
I20240503 08:05:48.795209 21438 tablet_copy_client.cc:308] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46199
I20240503 08:05:48.795701 19616 tablet_copy_service.cc:140] P 563200d2e6694145a13a4fd97d23b4f4: Received BeginTabletCopySession request for tablet bc872d506be04591a178b33ca06d09f2 from peer b38441293bcc4b57b5f4c09adeaf70ad ({username='slave'} at 127.0.99.129:43849)
I20240503 08:05:48.795809 19616 tablet_copy_service.cc:161] P 563200d2e6694145a13a4fd97d23b4f4: Beginning new tablet copy session on tablet bc872d506be04591a178b33ca06d09f2 from peer b38441293bcc4b57b5f4c09adeaf70ad at {username='slave'} at 127.0.99.129:43849: session id = b38441293bcc4b57b5f4c09adeaf70ad-bc872d506be04591a178b33ca06d09f2
I20240503 08:05:48.796479 19616 tablet_copy_source_session.cc:205] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:48.797106 21438 ts_tablet_manager.cc:1875] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:48.798810 21438 ts_tablet_manager.cc:1888] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:48.799000 21438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc872d506be04591a178b33ca06d09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:48.799778 21438 tablet_copy_client.cc:783] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:48.799917 21438 tablet_copy_client.cc:647] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:48.800895 21438 tablet_copy_client.cc:523] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:48.801751 21438 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap starting.
I20240503 08:05:48.818531 21438 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:48.818769 21438 tablet_bootstrap.cc:492] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Bootstrap complete.
I20240503 08:05:48.818910 21438 ts_tablet_manager.cc:1367] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240503 08:05:48.819295 21438 raft_consensus.cc:348] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.819420 21438 raft_consensus.cc:729] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b38441293bcc4b57b5f4c09adeaf70ad, State: Initialized, Role: LEARNER
I20240503 08:05:48.819504 21438 consensus_queue.cc:260] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: true } }
I20240503 08:05:48.819736 21438 ts_tablet_manager.cc:1398] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:48.820281 19616 tablet_copy_service.cc:342] P 563200d2e6694145a13a4fd97d23b4f4: Request end of tablet copy session b38441293bcc4b57b5f4c09adeaf70ad-bc872d506be04591a178b33ca06d09f2 received from {username='slave'} at 127.0.99.129:43849
I20240503 08:05:48.820374 19616 tablet_copy_service.cc:433] P 563200d2e6694145a13a4fd97d23b4f4: ending tablet copy session b38441293bcc4b57b5f4c09adeaf70ad-bc872d506be04591a178b33ca06d09f2 on tablet bc872d506be04591a178b33ca06d09f2 with peer b38441293bcc4b57b5f4c09adeaf70ad
I20240503 08:05:48.824470 21454 ts_tablet_manager.cc:902] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Initiating tablet copy from peer f835c24114954b93a22708d01227b666 (127.0.99.131:45203)
I20240503 08:05:48.825126 21454 tablet_copy_client.cc:308] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45203
I20240503 08:05:48.825634 19748 tablet_copy_service.cc:140] P f835c24114954b93a22708d01227b666: Received BeginTabletCopySession request for tablet 40ea96d08bcb46028cc7f907b489e76b from peer 563200d2e6694145a13a4fd97d23b4f4 ({username='slave'} at 127.0.99.130:52911)
I20240503 08:05:48.825729 19748 tablet_copy_service.cc:161] P f835c24114954b93a22708d01227b666: Beginning new tablet copy session on tablet 40ea96d08bcb46028cc7f907b489e76b from peer 563200d2e6694145a13a4fd97d23b4f4 at {username='slave'} at 127.0.99.130:52911: session id = 563200d2e6694145a13a4fd97d23b4f4-40ea96d08bcb46028cc7f907b489e76b
I20240503 08:05:48.826408 19748 tablet_copy_source_session.cc:205] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:48.827268 21454 ts_tablet_manager.cc:1875] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:48.829018 21454 ts_tablet_manager.cc:1888] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:48.829166 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ea96d08bcb46028cc7f907b489e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:48.830077 21454 tablet_copy_client.cc:783] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:48.830214 21454 tablet_copy_client.cc:647] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:48.831547 21454 tablet_copy_client.cc:523] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:48.832768 21454 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap starting.
I20240503 08:05:48.850844 21454 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:48.851142 21454 tablet_bootstrap.cc:492] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Bootstrap complete.
I20240503 08:05:48.851300 21454 ts_tablet_manager.cc:1367] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.001s
I20240503 08:05:48.851755 21454 raft_consensus.cc:348] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } }
I20240503 08:05:48.851867 21454 raft_consensus.cc:729] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 563200d2e6694145a13a4fd97d23b4f4, State: Initialized, Role: LEARNER
I20240503 08:05:48.851990 21454 consensus_queue.cc:260] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: true } }
I20240503 08:05:48.852239 21454 ts_tablet_manager.cc:1398] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:48.852828 19748 tablet_copy_service.cc:342] P f835c24114954b93a22708d01227b666: Request end of tablet copy session 563200d2e6694145a13a4fd97d23b4f4-40ea96d08bcb46028cc7f907b489e76b received from {username='slave'} at 127.0.99.130:52911
I20240503 08:05:48.852905 19748 tablet_copy_service.cc:433] P f835c24114954b93a22708d01227b666: ending tablet copy session 563200d2e6694145a13a4fd97d23b4f4-40ea96d08bcb46028cc7f907b489e76b on tablet 40ea96d08bcb46028cc7f907b489e76b with peer 563200d2e6694145a13a4fd97d23b4f4
I20240503 08:05:49.088438 19463 raft_consensus.cc:1203] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:05:49.316951 19463 raft_consensus.cc:1203] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:05:49.367300 19595 raft_consensus.cc:1203] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:05:49.494020 21476 raft_consensus.cc:1050] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704: attempting to promote NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad to VOTER
I20240503 08:05:49.494437 21476 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } }
I20240503 08:05:49.495770 19463 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 LEARNER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:49.495803 19726 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:49.495831 19595 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:49.496167 21476 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.496357 21386 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.496527 21512 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.498328 21386 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.498751 19463 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.499878 19595 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.500866 19726 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.501772 19319 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad reported cstate change: config changed from index 8 to 9, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ea59580caefd4780bb4f019a0133e704" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { replace: true } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.505440 20479 consensus_queue.cc:237] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } }
I20240503 08:05:49.506438 19726 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:49.506505 19463 raft_consensus.cc:1261] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Refusing update from remote peer ea59580caefd4780bb4f019a0133e704: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:05:49.506814 21476 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:49.507022 21512 consensus_queue.cc:1035] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:49.508483 21475 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P ea59580caefd4780bb4f019a0133e704 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.508915 19726 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.508975 19463 raft_consensus.cc:2932] T eb2dde0bfa18445d8b4a05db1030ef65 P b38441293bcc4b57b5f4c09adeaf70ad [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.510155 19306 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb2dde0bfa18445d8b4a05db1030ef65 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:49.511216 19320 catalog_manager.cc:5484] T eb2dde0bfa18445d8b4a05db1030ef65 P f835c24114954b93a22708d01227b666 reported cstate change: config changed from index 9 to 10, VOTER 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "ea59580caefd4780bb4f019a0133e704" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.513295 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2dde0bfa18445d8b4a05db1030ef65 with delete_type TABLET_DATA_TOMBSTONED (TS 563200d2e6694145a13a4fd97d23b4f4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38212
I20240503 08:05:49.513641 21520 tablet_replica.cc:330] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: stopping tablet replica
I20240503 08:05:49.513823 21520 raft_consensus.cc:2229] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:49.514156 21520 raft_consensus.cc:2258] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:49.514714 21520 ts_tablet_manager.cc:1875] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:49.516321 21520 ts_tablet_manager.cc:1888] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:05:49.516384 21520 log.cc:1196] T eb2dde0bfa18445d8b4a05db1030ef65 P 563200d2e6694145a13a4fd97d23b4f4: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb2dde0bfa18445d8b4a05db1030ef65
I20240503 08:05:49.516813 19307 catalog_manager.cc:4830] TS 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130:46199): tablet eb2dde0bfa18445d8b4a05db1030ef65 (table rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]) successfully deleted
I20240503 08:05:49.702800 21230 raft_consensus.cc:1050] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4: attempting to promote NON_VOTER b38441293bcc4b57b5f4c09adeaf70ad to VOTER
I20240503 08:05:49.703225 21230 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } }
I20240503 08:05:49.704463 19463 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 LEARNER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.704521 19726 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.704558 20479 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.704881 21493 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.705071 21547 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.705235 21492 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.707235 21492 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.707686 19463 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.707659 19726 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.709699 20479 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.710086 19316 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 7 to 8, b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:49.712782 19595 consensus_queue.cc:237] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } }
I20240503 08:05:49.713878 19463 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:49.714036 20479 raft_consensus.cc:1261] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Refusing update from remote peer 563200d2e6694145a13a4fd97d23b4f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:49.714300 21493 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.714463 21492 consensus_queue.cc:1035] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.715958 21547 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.716396 20479 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P ea59580caefd4780bb4f019a0133e704 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.716477 19463 raft_consensus.cc:2932] T bc872d506be04591a178b33ca06d09f2 P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } } }
I20240503 08:05:49.717509 19307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc872d506be04591a178b33ca06d09f2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:49.718036 19319 catalog_manager.cc:5484] T bc872d506be04591a178b33ca06d09f2 P 563200d2e6694145a13a4fd97d23b4f4 reported cstate change: config changed from index 8 to 9, VOTER f835c24114954b93a22708d01227b666 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "563200d2e6694145a13a4fd97d23b4f4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea59580caefd4780bb4f019a0133e704" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:49.720436 19707 tablet_service.cc:1514] Processing DeleteTablet for tablet bc872d506be04591a178b33ca06d09f2 with delete_type TABLET_DATA_TOMBSTONED (TS f835c24114954b93a22708d01227b666 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55674
I20240503 08:05:49.720953 21551 tablet_replica.cc:330] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: stopping tablet replica
I20240503 08:05:49.721093 21551 raft_consensus.cc:2229] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:49.721314 21551 raft_consensus.cc:2258] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:49.721932 21551 ts_tablet_manager.cc:1875] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:49.723631 21551 ts_tablet_manager.cc:1888] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:05:49.723693 21551 log.cc:1196] T bc872d506be04591a178b33ca06d09f2 P f835c24114954b93a22708d01227b666: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc872d506be04591a178b33ca06d09f2
I20240503 08:05:49.724110 19306 catalog_manager.cc:4830] TS f835c24114954b93a22708d01227b666 (127.0.99.131:45203): tablet bc872d506be04591a178b33ca06d09f2 (table rebalance_test_table_1 [id=7e4c3e4e801049268200a3f57b0bb41c]) successfully deleted
I20240503 08:05:49.944922 21510 raft_consensus.cc:1050] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666: attempting to promote NON_VOTER 563200d2e6694145a13a4fd97d23b4f4 to VOTER
I20240503 08:05:49.945315 21510 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } }
I20240503 08:05:49.946703 20347 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.946722 19463 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.946714 19595 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 LEARNER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:49.947212 21510 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.947422 21568 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.947595 21569 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:49.949471 21569 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.949924 20347 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.951664 19463 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.951838 19595 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.953393 19321 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb reported cstate change: config changed from index 7 to 8, 563200d2e6694145a13a4fd97d23b4f4 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "b38441293bcc4b57b5f4c09adeaf70ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44963 } attrs { replace: true } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.956650 19726 consensus_queue.cc:237] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } }
I20240503 08:05:49.957626 19595 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:49.957795 20347 raft_consensus.cc:1261] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Refusing update from remote peer f835c24114954b93a22708d01227b666: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:49.958153 21568 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.958343 21510 consensus_queue.cc:1035] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:49.959810 21569 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.960228 19595 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 563200d2e6694145a13a4fd97d23b4f4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.960211 20347 raft_consensus.cc:2932] T 40ea96d08bcb46028cc7f907b489e76b P 2d709df031144ec6b5c0fd04315bd1fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } } }
I20240503 08:05:49.961823 19306 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40ea96d08bcb46028cc7f907b489e76b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:49.962003 19320 catalog_manager.cc:5484] T 40ea96d08bcb46028cc7f907b489e76b P f835c24114954b93a22708d01227b666 reported cstate change: config changed from index 8 to 9, VOTER b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "f835c24114954b93a22708d01227b666" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f835c24114954b93a22708d01227b666" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d709df031144ec6b5c0fd04315bd1fb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563200d2e6694145a13a4fd97d23b4f4" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46199 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:49.964284 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ea96d08bcb46028cc7f907b489e76b with delete_type TABLET_DATA_TOMBSTONED (TS b38441293bcc4b57b5f4c09adeaf70ad not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37106
I20240503 08:05:49.964601 21574 tablet_replica.cc:330] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: stopping tablet replica
I20240503 08:05:49.964754 21574 raft_consensus.cc:2229] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:49.964977 21574 raft_consensus.cc:2258] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:49.965605 21574 ts_tablet_manager.cc:1875] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:49.967267 21574 ts_tablet_manager.cc:1888] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:05:49.967330 21574 log.cc:1196] T 40ea96d08bcb46028cc7f907b489e76b P b38441293bcc4b57b5f4c09adeaf70ad: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/40ea96d08bcb46028cc7f907b489e76b
I20240503 08:05:49.967739 19307 catalog_manager.cc:4830] TS b38441293bcc4b57b5f4c09adeaf70ad (127.0.99.129:44963): tablet 40ea96d08bcb46028cc7f907b489e76b (table rebalance_test_table_0 [id=79b8f977559f4482a35cc2bee24506e5]) successfully deleted
I20240503 08:05:50.045677 19576 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:50.045945 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:50.046731 20459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:50.057231 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:50.064339 19442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:50.135500 19576 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:05:50.135529 20328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:50.135655 19442 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:50.139092 19707 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:50.151396 20459 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:50.181555 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19382
I20240503 08:05:50.188052 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19514
I20240503 08:05:50.193895 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19645
I20240503 08:05:50.199611 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 20397
I20240503 08:05:50.204385 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 20266
I20240503 08:05:50.209590 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 19291
2024-05-03T08:05:50Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (14949 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240503 08:05:50.234771 398 test_util.cc:268] Using random seed: 806979723
I20240503 08:05:50.235662 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:05:50.235707 398 ts_itest-base.cc:113] --------------
I20240503 08:05:50.235723 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:05:50.235735 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:05:50.235749 398 ts_itest-base.cc:116] --------------
2024-05-03T08:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:05:50Z Disabled control of system clock
I20240503 08:05:50.244189 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43583
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43583
--location_mapping_cmd=/tmp/dist-test-task8ZGZ68/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:05:50.346589 21644 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:05:50.346849 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:50.346875 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:50.346938 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:50.349944 21644 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:50.350021 21644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:05:50.350049 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:50.350064 21644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:05:50.350080 21644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:05:50.353930 21644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task8ZGZ68/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43583
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:50.354831 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:50.355988 21644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:50.363399 21649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.364372 21652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.364467 21650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:50.364905 21644 server_base.cc:1034] running on GCE node
I20240503 08:05:50.365397 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:50.366334 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:50.367491 21644 hybrid_clock.cc:648] HybridClock initialized: now 1714723550367469 us; error 40 us; skew 500 ppm
I20240503 08:05:50.369346 21644 webserver.cc:460] Webserver started at http://127.0.99.190:40005/ using document root <none> and password file <none>
I20240503 08:05:50.369894 21644 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:50.369949 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:50.370112 21644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:50.371788 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4c06655df91b4c73bd8c7d8a44ddcef5"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.372232 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4c06655df91b4c73bd8c7d8a44ddcef5"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.375401 21644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:05:50.377481 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.378451 21644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:50.378567 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4c06655df91b4c73bd8c7d8a44ddcef5"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.378645 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:50.390457 21644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:50.391134 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:50.391279 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:50.397673 21644 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43583
I20240503 08:05:50.397739 21710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43583 every 8 connection(s)
I20240503 08:05:50.398622 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:05:50.401051 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:50.405882 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5: Bootstrap starting.
I20240503 08:05:50.408051 21711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:50.408486 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 21644
I20240503 08:05:50.408614 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:05:50.408752 21711 log.cc:824] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:50.410485 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5: No bootstrap required, opened a new log
I20240503 08:05:50.412860 21711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } }
I20240503 08:05:50.413028 21711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:50.413080 21711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c06655df91b4c73bd8c7d8a44ddcef5, State: Initialized, Role: FOLLOWER
I20240503 08:05:50.413448 21711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } }
I20240503 08:05:50.413559 21711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:05:50.413614 21711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:05:50.413724 21711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:50.414747 21711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } }
I20240503 08:05:50.415118 21711 leader_election.cc:302] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c06655df91b4c73bd8c7d8a44ddcef5; no voters:
I20240503 08:05:50.415380 21711 leader_election.cc:288] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:05:50.415447 21716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:50.415616 21716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [term 1 LEADER]: Becoming Leader. State: Replica: 4c06655df91b4c73bd8c7d8a44ddcef5, State: Running, Role: LEADER
I20240503 08:05:50.415882 21716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } }
I20240503 08:05:50.416461 21711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:05:50.417388 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } } }
I20240503 08:05:50.417418 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c06655df91b4c73bd8c7d8a44ddcef5. Latest consensus state: current_term: 1 leader_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c06655df91b4c73bd8c7d8a44ddcef5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43583 } } }
I20240503 08:05:50.417527 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:50.417531 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:05:50.417840 21725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:05:50.420624 21725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:05:50.425741 21725 catalog_manager.cc:1347] Generated new cluster ID: beb63f0ec4194a7682c3f90cac88d3ba
I20240503 08:05:50.425805 21725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:05:50.441840 21725 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:05:50.442775 21725 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:05:50.460160 21725 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4c06655df91b4c73bd8c7d8a44ddcef5: Generated new TSK 0
I20240503 08:05:50.460904 21725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:05:50.467679 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:05:50.570441 21735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:50.570644 21735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:50.570685 21735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:50.573735 21735 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:50.573794 21735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:50.573860 21735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:05:50.577399 21735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--never_fsync=true
--heap_profile_path=/tmp/kudu.21735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:50.578212 21735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:50.579277 21735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:50.586947 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.587204 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.587587 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:50.587987 21735 server_base.cc:1034] running on GCE node
I20240503 08:05:50.588322 21735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:50.588836 21735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:50.589970 21735 hybrid_clock.cc:648] HybridClock initialized: now 1714723550589952 us; error 38 us; skew 500 ppm
I20240503 08:05:50.591771 21735 webserver.cc:460] Webserver started at http://127.0.99.129:36103/ using document root <none> and password file <none>
I20240503 08:05:50.592303 21735 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:50.592360 21735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:50.592540 21735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:50.594103 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5de0b16d88c64038b34727ac7f94d9ee"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.594545 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5de0b16d88c64038b34727ac7f94d9ee"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.597646 21735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:05:50.599661 21749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.600622 21735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:50.600716 21735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5de0b16d88c64038b34727ac7f94d9ee"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.600790 21735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:50.614907 21735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:50.615507 21735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:50.615656 21735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:50.616161 21735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:50.617044 21735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:50.617089 21735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.617139 21735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:50.617167 21735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.625054 21735 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33957
I20240503 08:05:50.625087 21862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33957 every 8 connection(s)
I20240503 08:05:50.625916 21735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:05:50.632998 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 21735
I20240503 08:05:50.633127 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:05:50.633985 21863 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43583
I20240503 08:05:50.634199 21863 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:50.634765 21863 heartbeater.cc:511] Master 127.0.99.190:43583 requested a full tablet report, sending...
I20240503 08:05:50.635039 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:50.684724 21675 ts_manager.cc:194] Registered new tserver with Master: 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:50.686025 21675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:40549
W20240503 08:05:50.751570 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:50.751791 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:50.751839 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:50.755012 21866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:50.755077 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:50.755165 21866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:05:50.758797 21866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--never_fsync=true
--heap_profile_path=/tmp/kudu.21866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:50.759688 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:50.760735 21866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:50.768321 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.768687 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.769050 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:50.769450 21866 server_base.cc:1034] running on GCE node
I20240503 08:05:50.769874 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:50.770395 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:50.771535 21866 hybrid_clock.cc:648] HybridClock initialized: now 1714723550771517 us; error 39 us; skew 500 ppm
I20240503 08:05:50.773293 21866 webserver.cc:460] Webserver started at http://127.0.99.130:36599/ using document root <none> and password file <none>
I20240503 08:05:50.773816 21866 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:50.773871 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:50.774031 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:50.775624 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ddaa7bb712a245b4a37a1c511049924d"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.776064 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ddaa7bb712a245b4a37a1c511049924d"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.779178 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:05:50.781255 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.782302 21866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:50.782398 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ddaa7bb712a245b4a37a1c511049924d"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.782474 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:50.793303 21866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:50.793860 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:50.794005 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:50.794492 21866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:50.795384 21866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:50.795429 21866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.795480 21866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:50.795503 21866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.803926 21866 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:44025
I20240503 08:05:50.803953 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:44025 every 8 connection(s)
I20240503 08:05:50.804912 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:05:50.810431 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 21866
I20240503 08:05:50.810555 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:05:50.812824 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:50.813694 21995 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43583
I20240503 08:05:50.813938 21995 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:50.814458 21995 heartbeater.cc:511] Master 127.0.99.190:43583 requested a full tablet report, sending...
I20240503 08:05:50.854317 21675 ts_manager.cc:194] Registered new tserver with Master: ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:50.855248 21675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56719
W20240503 08:05:50.919493 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:50.919703 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:50.919747 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:50.922777 21998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:50.922837 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:50.922904 21998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:05:50.926470 21998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--never_fsync=true
--heap_profile_path=/tmp/kudu.21998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:50.927371 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:50.928457 21998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:50.936148 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.936492 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:50.936904 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:50.937290 21998 server_base.cc:1034] running on GCE node
I20240503 08:05:50.937628 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:50.938172 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:50.939307 21998 hybrid_clock.cc:648] HybridClock initialized: now 1714723550939292 us; error 30 us; skew 500 ppm
I20240503 08:05:50.941076 21998 webserver.cc:460] Webserver started at http://127.0.99.131:33861/ using document root <none> and password file <none>
I20240503 08:05:50.941604 21998 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:50.941656 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:50.941833 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:50.943464 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "63c7e61c2efa45ebae4113ff375dc22c"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.943936 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "63c7e61c2efa45ebae4113ff375dc22c"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.947139 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:05:50.949149 22013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.950109 21998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:50.950202 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "63c7e61c2efa45ebae4113ff375dc22c"
format_stamp: "Formatted at 2024-05-03 08:05:50 on dist-test-slave-11df"
I20240503 08:05:50.950284 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:50.974761 21998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:50.975422 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:50.975575 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:50.976120 21998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:50.977016 21998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:50.977061 21998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.977103 21998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:50.977131 21998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:50.985270 21998 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:35301
I20240503 08:05:50.985306 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:35301 every 8 connection(s)
I20240503 08:05:50.986207 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:05:50.988278 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 21998
I20240503 08:05:50.988394 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:05:50.990423 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:05:50.995121 22127 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43583
I20240503 08:05:50.995338 22127 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:50.995846 22127 heartbeater.cc:511] Master 127.0.99.190:43583 requested a full tablet report, sending...
I20240503 08:05:51.031422 21675 ts_manager.cc:194] Registered new tserver with Master: 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:51.032119 21675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47319
W20240503 08:05:51.096657 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:51.096859 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:51.096906 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:51.099905 22130 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:51.099962 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:51.100039 22130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:51.103593 22130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43583
--never_fsync=true
--heap_profile_path=/tmp/kudu.22130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:51.104532 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:51.105644 22130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:51.113233 22136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:51.113539 22139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:51.113945 22137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:51.114329 22130 server_base.cc:1034] running on GCE node
I20240503 08:05:51.114670 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:51.115252 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:51.116385 22130 hybrid_clock.cc:648] HybridClock initialized: now 1714723551116368 us; error 31 us; skew 500 ppm
I20240503 08:05:51.118371 22130 webserver.cc:460] Webserver started at http://127.0.99.132:34229/ using document root <none> and password file <none>
I20240503 08:05:51.118935 22130 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:51.119004 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:51.119184 22130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:05:51.120800 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e56f5071860c4210a63b8241a2233f00"
format_stamp: "Formatted at 2024-05-03 08:05:51 on dist-test-slave-11df"
I20240503 08:05:51.121282 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e56f5071860c4210a63b8241a2233f00"
format_stamp: "Formatted at 2024-05-03 08:05:51 on dist-test-slave-11df"
I20240503 08:05:51.124562 22130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:05:51.126634 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:51.127774 22130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:05:51.127877 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e56f5071860c4210a63b8241a2233f00"
format_stamp: "Formatted at 2024-05-03 08:05:51 on dist-test-slave-11df"
I20240503 08:05:51.127961 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:51.140527 22130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:51.141142 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:51.141310 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:51.141839 22130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:51.142717 22130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:51.142772 22130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:51.142822 22130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:51.142845 22130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:51.150525 22130 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45789
I20240503 08:05:51.150564 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45789 every 8 connection(s)
I20240503 08:05:51.151470 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:51.155643 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 22130
I20240503 08:05:51.155757 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:05:51.159986 22259 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43583
I20240503 08:05:51.160178 22259 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:51.160640 22259 heartbeater.cc:511] Master 127.0.99.190:43583 requested a full tablet report, sending...
I20240503 08:05:51.193815 21675 ts_manager.cc:194] Registered new tserver with Master: e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789)
I20240503 08:05:51.194463 21675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60107
I20240503 08:05:51.205662 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:05:51.214722 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:05:51.214808 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu with pid 22130
I20240503 08:05:51.688390 21863 heartbeater.cc:503] Master 127.0.99.190:43583 was elected leader, sending a full tablet report...
I20240503 08:05:51.857452 21995 heartbeater.cc:503] Master 127.0.99.190:43583 was elected leader, sending a full tablet report...
I20240503 08:05:52.034582 22127 heartbeater.cc:503] Master 127.0.99.190:43583 was elected leader, sending a full tablet report...
I20240503 08:05:54.971801 21675 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:54.975355 21675 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:54.992942 21797 tablet_service.cc:1467] Processing CreateTablet for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:54.992947 21929 tablet_service.cc:1467] Processing CreateTablet for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:54.992947 21795 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:54.993734 21796 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1ade447f7c4a16ae3336b3d0292233 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:54.993886 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3eb0eda0be496e9450d98e8a1f5b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:54.993887 21929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:54.994228 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1ade447f7c4a16ae3336b3d0292233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:54.995155 21797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.000098 21928 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1ade447f7c4a16ae3336b3d0292233 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.000358 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1ade447f7c4a16ae3336b3d0292233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.002447 22279 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.002910 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.003158 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3eb0eda0be496e9450d98e8a1f5b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.004649 22279 tablet_bootstrap.cc:654] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.005455 22279 log.cc:824] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:55.006840 22061 tablet_service.cc:1467] Processing CreateTablet for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.007189 22059 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.007813 22061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.007838 22060 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1ade447f7c4a16ae3336b3d0292233 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.008117 22059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3eb0eda0be496e9450d98e8a1f5b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.008422 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1ade447f7c4a16ae3336b3d0292233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.009502 22279 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.009676 22279 ts_tablet_manager.cc:1367] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:05:55.010587 22278 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.012719 22278 tablet_bootstrap.cc:654] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.013157 22279 raft_consensus.cc:348] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.013384 22279 raft_consensus.cc:374] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.013479 22279 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.013515 22278 log.cc:824] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:55.013965 22279 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.014864 22279 ts_tablet_manager.cc:1398] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 08:05:55.015148 22279 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.015615 22281 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.015924 22279 tablet_bootstrap.cc:654] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.017148 22279 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.017287 22279 ts_tablet_manager.cc:1367] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:55.017307 22281 tablet_bootstrap.cc:654] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.017859 22279 raft_consensus.cc:348] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.017953 22279 raft_consensus.cc:374] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.018075 22279 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.018086 22281 log.cc:824] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:55.018226 22279 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.018528 22279 ts_tablet_manager.cc:1398] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.018724 22279 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.019593 22279 tablet_bootstrap.cc:654] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.019865 22281 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.019865 22278 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.020030 22278 ts_tablet_manager.cc:1367] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:05:55.020030 22281 ts_tablet_manager.cc:1367] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:05:55.020665 22279 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.020733 22279 ts_tablet_manager.cc:1367] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.021164 22279 raft_consensus.cc:348] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.021251 22279 raft_consensus.cc:374] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.021281 22279 raft_consensus.cc:729] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.021390 22279 consensus_queue.cc:260] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.021638 22279 ts_tablet_manager.cc:1398] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.023272 22278 raft_consensus.cc:348] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.023478 22278 raft_consensus.cc:374] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.023567 22278 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.023545 22281 raft_consensus.cc:348] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.023717 22281 raft_consensus.cc:374] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.023769 22281 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.024029 22278 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.024109 22281 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.024886 22281 ts_tablet_manager.cc:1398] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:05:55.024886 22278 ts_tablet_manager.cc:1398] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:05:55.025158 22278 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.025158 22281 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.026018 22278 tablet_bootstrap.cc:654] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.026018 22281 tablet_bootstrap.cc:654] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.027194 22278 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.027285 22281 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.027292 22278 ts_tablet_manager.cc:1367] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:55.027364 22281 ts_tablet_manager.cc:1367] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.027779 22281 raft_consensus.cc:348] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.027858 22281 raft_consensus.cc:374] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.027886 22281 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.027858 22278 raft_consensus.cc:348] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.027998 22281 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.028038 22278 raft_consensus.cc:374] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.028085 22278 raft_consensus.cc:729] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.028246 22281 ts_tablet_manager.cc:1398] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.028249 22278 consensus_queue.cc:260] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.028395 22281 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.028558 22278 ts_tablet_manager.cc:1398] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.028753 22278 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.029139 22281 tablet_bootstrap.cc:654] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.030179 22281 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.030251 22281 ts_tablet_manager.cc:1367] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.030580 22278 tablet_bootstrap.cc:654] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.030673 22281 raft_consensus.cc:348] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.030766 22281 raft_consensus.cc:374] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.030797 22281 raft_consensus.cc:729] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.030900 22281 consensus_queue.cc:260] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.031145 22281 ts_tablet_manager.cc:1398] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.031883 22278 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.031963 22278 ts_tablet_manager.cc:1367] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:05:55.032434 22278 raft_consensus.cc:348] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.032552 22278 raft_consensus.cc:374] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.032594 22278 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.032738 22278 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.033013 22278 ts_tablet_manager.cc:1398] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:55.033291 22288 raft_consensus.cc:482] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.033427 22288 raft_consensus.cc:504] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.034415 22288 leader_election.cc:288] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:55.038460 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:55.038740 21817 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.039134 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.039146 22015 leader_election.cc:302] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, 63c7e61c2efa45ebae4113ff375dc22c; no voters:
I20240503 08:05:55.039346 22288 raft_consensus.cc:2781] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.039419 22288 raft_consensus.cc:482] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.039374 21948 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.039469 22288 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.040393 22288 raft_consensus.cc:504] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.040689 22288 leader_election.cc:288] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:55.040977 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:55.041057 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.041082 21817 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.041179 21948 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.042223 21817 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.042272 21948 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.042544 22015 leader_election.cc:302] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, 63c7e61c2efa45ebae4113ff375dc22c; no voters:
I20240503 08:05:55.042739 22288 raft_consensus.cc:2781] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.042949 22288 raft_consensus.cc:686] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:55.043263 22288 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.045994 21675 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } }
W20240503 08:05:55.058430 21996 tablet.cc:2369] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:05:55.075778 22288 raft_consensus.cc:482] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.075922 22288 raft_consensus.cc:504] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.076331 22288 leader_election.cc:288] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.076754 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54dfa529ecc4aa2ae4056ed6c8fe2c3" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.076789 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54dfa529ecc4aa2ae4056ed6c8fe2c3" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:55.076892 21948 raft_consensus.cc:2454] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.076910 21817 raft_consensus.cc:2454] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.077199 22016 leader_election.cc:302] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.077359 22288 raft_consensus.cc:2781] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.077418 22288 raft_consensus.cc:482] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.077464 22288 raft_consensus.cc:3037] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.078233 22288 raft_consensus.cc:504] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.078555 22288 leader_election.cc:288] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.078869 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54dfa529ecc4aa2ae4056ed6c8fe2c3" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.078864 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54dfa529ecc4aa2ae4056ed6c8fe2c3" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:55.078994 21948 raft_consensus.cc:3037] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.078998 21817 raft_consensus.cc:3037] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.079692 21948 raft_consensus.cc:2454] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.079699 21817 raft_consensus.cc:2454] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.079949 22016 leader_election.cc:302] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.080112 22288 raft_consensus.cc:2781] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.080195 22288 raft_consensus.cc:686] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:55.080338 22288 consensus_queue.cc:237] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:05:55.081871 21675 catalog_manager.cc:5484] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:55.094993 22288 raft_consensus.cc:482] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.095124 22288 raft_consensus.cc:504] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.095528 22288 leader_election.cc:288] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.095937 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:55.095942 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.096069 21948 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.096078 21817 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.096344 22016 leader_election.cc:302] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.096511 22288 raft_consensus.cc:2781] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.096562 22288 raft_consensus.cc:482] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.096587 22288 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.097301 22288 raft_consensus.cc:504] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.097607 22288 leader_election.cc:288] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.097924 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.097927 21817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:55.098029 21948 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.098049 21817 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.098702 21817 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.098726 21948 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.098982 22016 leader_election.cc:302] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.099133 22288 raft_consensus.cc:2781] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.099220 22288 raft_consensus.cc:686] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:55.099503 22288 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.101009 21675 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } }
W20240503 08:05:55.130007 21864 tablet.cc:2369] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:05:55.213941 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:55.214190 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:55.217362 22299 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 08:05:55.239842 22128 tablet.cc:2369] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Using table 'rebalance_test_table_0'
I20240503 08:05:55.312155 21817 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.313223 21816 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.314561 21948 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.315068 21949 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.315469 22295 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.314568 21947 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.315608 22288 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.315929 22295 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.316228 22329 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.316279 22328 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.317333 21817 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.317768 22329 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.323454 22326 mvcc.cc:205] Tried to move back new op lower bound from 7023507682572132352 to 7023507681610309632. Current Snapshot: MvccSnapshot[applied={T|T < 7023507682555211776}]
I20240503 08:05:55.334137 22331 mvcc.cc:205] Tried to move back new op lower bound from 7023507682572132352 to 7023507681610309632. Current Snapshot: MvccSnapshot[applied={T|T < 7023507682555211776}]
INSERT report
rows total: 2000
time total: 101.435 ms
time per row: 0.0507177 ms
I20240503 08:05:55.358994 22340 mvcc.cc:205] Tried to move back new op lower bound from 7023507682569236480 to 7023507681688903680. Current Snapshot: MvccSnapshot[applied={T|T < 7023507682551918592}]
I20240503 08:05:55.369532 21667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:55.370059 21667 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:55.375084 21929 tablet_service.cc:1467] Processing CreateTablet for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.375360 21929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8260d37c2084d6a9bd2f3c6ac7e5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.375293 22061 tablet_service.cc:1467] Processing CreateTablet for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.375545 22061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8260d37c2084d6a9bd2f3c6ac7e5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.375515 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 210ed86636fb4c1584cb09fc63598447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.375787 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210ed86636fb4c1584cb09fc63598447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.375813 21796 tablet_service.cc:1467] Processing CreateTablet for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.376053 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8260d37c2084d6a9bd2f3c6ac7e5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.376994 22059 tablet_service.cc:1467] Processing CreateTablet for tablet 210ed86636fb4c1584cb09fc63598447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.377213 22059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210ed86636fb4c1584cb09fc63598447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.377470 22281 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.378069 21795 tablet_service.cc:1467] Processing CreateTablet for tablet 210ed86636fb4c1584cb09fc63598447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.378194 21797 tablet_service.cc:1467] Processing CreateTablet for tablet 59ddd92adecf4d75a6cd209cb59788fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.378270 22281 tablet_bootstrap.cc:654] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.378288 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210ed86636fb4c1584cb09fc63598447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.378405 21797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ddd92adecf4d75a6cd209cb59788fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.379093 22060 tablet_service.cc:1467] Processing CreateTablet for tablet 59ddd92adecf4d75a6cd209cb59788fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.379278 22281 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.379297 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ddd92adecf4d75a6cd209cb59788fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.379350 22281 ts_tablet_manager.cc:1367] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.379767 22281 raft_consensus.cc:348] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.379865 22281 raft_consensus.cc:374] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.379892 22281 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.380009 22281 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.380257 22281 ts_tablet_manager.cc:1398] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.380380 22281 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.380949 22279 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.381213 22281 tablet_bootstrap.cc:654] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.381721 22279 tablet_bootstrap.cc:654] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.381726 22278 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.382426 22278 tablet_bootstrap.cc:654] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.382660 22279 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.382683 22281 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.382748 22279 ts_tablet_manager.cc:1367] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.382755 22281 ts_tablet_manager.cc:1367] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.383167 22281 raft_consensus.cc:348] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.383190 22279 raft_consensus.cc:348] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.383270 22281 raft_consensus.cc:374] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.383293 22279 raft_consensus.cc:374] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.383301 22281 raft_consensus.cc:729] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.383325 22279 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.383419 22281 consensus_queue.cc:260] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.383440 22279 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.383682 22279 ts_tablet_manager.cc:1398] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.383750 22281 ts_tablet_manager.cc:1398] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.383836 22279 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.383895 22281 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:55.384040 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 59ddd92adecf4d75a6cd209cb59788fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:55.384248 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ddd92adecf4d75a6cd209cb59788fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:55.384603 22279 tablet_bootstrap.cc:654] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.384665 22281 tablet_bootstrap.cc:654] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.385896 22281 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:55.385978 22281 ts_tablet_manager.cc:1367] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:55.386368 22281 raft_consensus.cc:348] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.386474 22281 raft_consensus.cc:374] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.386505 22281 raft_consensus.cc:729] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.386623 22279 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.386619 22281 consensus_queue.cc:260] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.386705 22279 ts_tablet_manager.cc:1367] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:05:55.386857 22281 ts_tablet_manager.cc:1398] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.387149 22279 raft_consensus.cc:348] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.387243 22279 raft_consensus.cc:374] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.387274 22279 raft_consensus.cc:729] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.387374 22279 consensus_queue.cc:260] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.388643 22278 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.388731 22278 ts_tablet_manager.cc:1367] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:05:55.389137 22278 raft_consensus.cc:348] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.389168 22279 ts_tablet_manager.cc:1398] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.389225 22278 raft_consensus.cc:374] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.389256 22278 raft_consensus.cc:729] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.389370 22278 consensus_queue.cc:260] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.389603 22278 ts_tablet_manager.cc:1398] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:55.389622 22279 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:55.389766 22278 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.390462 22278 tablet_bootstrap.cc:654] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.390653 22279 tablet_bootstrap.cc:654] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.391799 22278 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.391799 22279 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:55.391871 22279 ts_tablet_manager.cc:1367] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:55.391871 22278 ts_tablet_manager.cc:1367] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.392264 22278 raft_consensus.cc:348] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.392264 22279 raft_consensus.cc:348] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.392365 22279 raft_consensus.cc:374] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.392365 22278 raft_consensus.cc:374] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.392396 22279 raft_consensus.cc:729] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.392405 22278 raft_consensus.cc:729] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.392508 22279 consensus_queue.cc:260] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.392514 22278 consensus_queue.cc:260] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.392761 22278 ts_tablet_manager.cc:1398] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:55.392758 22279 ts_tablet_manager.cc:1398] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:55.392908 22278 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:55.393921 22278 tablet_bootstrap.cc:654] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:55.395095 22278 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:55.395171 22278 ts_tablet_manager.cc:1367] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:55.395494 22278 raft_consensus.cc:348] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.395567 22278 raft_consensus.cc:374] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:55.395596 22278 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:55.395709 22278 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.395916 22278 ts_tablet_manager.cc:1398] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:55.447070 22329 raft_consensus.cc:482] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.447222 22329 raft_consensus.cc:504] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.447655 22329 leader_election.cc:288] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.448084 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.448084 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:55.448218 21949 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.448222 21816 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.448518 22016 leader_election.cc:302] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.448688 22329 raft_consensus.cc:2781] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.448743 22329 raft_consensus.cc:482] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.448781 22329 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.449611 22329 raft_consensus.cc:504] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.449926 22329 leader_election.cc:288] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:55.450271 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.450282 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:55.450382 21816 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.450382 21949 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.451228 21949 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.451236 21816 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.451503 22016 leader_election.cc:302] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.451649 22329 raft_consensus.cc:2781] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.451735 22329 raft_consensus.cc:686] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:55.451869 22329 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.453513 21671 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } }
I20240503 08:05:55.495105 22330 raft_consensus.cc:482] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.495260 22330 raft_consensus.cc:504] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.496289 22330 leader_election.cc:288] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:55.499739 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210ed86636fb4c1584cb09fc63598447" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.499907 21949 raft_consensus.cc:2454] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 0.
I20240503 08:05:55.500187 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210ed86636fb4c1584cb09fc63598447" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c" is_pre_election: true
I20240503 08:05:55.500216 21752 leader_election.cc:302] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.500383 22081 raft_consensus.cc:2454] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 0.
I20240503 08:05:55.500452 22330 raft_consensus.cc:2781] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.500514 22330 raft_consensus.cc:482] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.500556 22330 raft_consensus.cc:3037] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.501372 22330 raft_consensus.cc:504] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.501709 22330 leader_election.cc:288] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:55.501984 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210ed86636fb4c1584cb09fc63598447" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.502094 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "210ed86636fb4c1584cb09fc63598447" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:05:55.502130 21949 raft_consensus.cc:3037] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.502207 22081 raft_consensus.cc:3037] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.503078 21949 raft_consensus.cc:2454] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 1.
I20240503 08:05:55.503141 22081 raft_consensus.cc:2454] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 1.
I20240503 08:05:55.503366 21752 leader_election.cc:302] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.503527 22330 raft_consensus.cc:2781] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.503614 22330 raft_consensus.cc:686] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Becoming Leader. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Running, Role: LEADER
I20240503 08:05:55.503875 22330 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.506132 21671 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:55.706207 22329 raft_consensus.cc:482] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:55.706357 22329 raft_consensus.cc:504] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.706828 22329 leader_election.cc:288] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:55.707329 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ddd92adecf4d75a6cd209cb59788fc" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:55.707334 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ddd92adecf4d75a6cd209cb59788fc" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:55.707461 21816 raft_consensus.cc:2454] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.707477 21949 raft_consensus.cc:2454] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:55.707772 22015 leader_election.cc:302] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, 63c7e61c2efa45ebae4113ff375dc22c; no voters:
I20240503 08:05:55.707934 22329 raft_consensus.cc:2781] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:55.707995 22329 raft_consensus.cc:482] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:55.708025 22329 raft_consensus.cc:3037] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.708804 22329 raft_consensus.cc:504] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.709100 22329 leader_election.cc:288] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:55.709424 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ddd92adecf4d75a6cd209cb59788fc" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:55.709435 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ddd92adecf4d75a6cd209cb59788fc" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:55.709527 21949 raft_consensus.cc:3037] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.709534 21816 raft_consensus.cc:3037] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:55.710212 21949 raft_consensus.cc:2454] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.710211 21816 raft_consensus.cc:2454] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:55.710461 22016 leader_election.cc:302] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:55.710647 22329 raft_consensus.cc:2781] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:55.710731 22329 raft_consensus.cc:686] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:55.710865 22329 consensus_queue.cc:237] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:55.712436 21671 catalog_manager.cc:5484] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } }
I20240503 08:05:55.834949 22329 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.839551 22288 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:05:55.851713 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:55.851974 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:55.855156 22370 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:05:55.892187 22330 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.896476 22287 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.943167 21949 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.943619 22295 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:55.948858 21817 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:55.952044 22329 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.156 ms
time per row: 0.0555781 ms
I20240503 08:05:56.011195 21669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:05:56.011729 21669 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:05:56.016742 21796 tablet_service.cc:1467] Processing CreateTablet for tablet 2d21167b87c04350b636468cdd81568f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.017045 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d21167b87c04350b636468cdd81568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.017619 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 2d21167b87c04350b636468cdd81568f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.017724 21929 tablet_service.cc:1467] Processing CreateTablet for tablet 5af6cecbb7f4422cb6c76239cf0bc002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.017891 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d21167b87c04350b636468cdd81568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.018033 21929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af6cecbb7f4422cb6c76239cf0bc002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.019403 21928 tablet_service.cc:1467] Processing CreateTablet for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.019605 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe7c2f06f184762aea5fe0dfd51d9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.020044 22425 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:56.020918 22425 tablet_bootstrap.cc:654] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.021654 22060 tablet_service.cc:1467] Processing CreateTablet for tablet 2d21167b87c04350b636468cdd81568f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.021888 22424 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:56.021915 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d21167b87c04350b636468cdd81568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.021948 21796 tablet_service.cc:1467] Processing CreateTablet for tablet 5af6cecbb7f4422cb6c76239cf0bc002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.021953 22059 tablet_service.cc:1467] Processing CreateTablet for tablet 5af6cecbb7f4422cb6c76239cf0bc002 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.022135 22059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af6cecbb7f4422cb6c76239cf0bc002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.022145 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af6cecbb7f4422cb6c76239cf0bc002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.022707 22424 tablet_bootstrap.cc:654] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.022881 22061 tablet_service.cc:1467] Processing CreateTablet for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.023118 22061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe7c2f06f184762aea5fe0dfd51d9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.023116 21797 tablet_service.cc:1467] Processing CreateTablet for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:05:56.023314 21797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe7c2f06f184762aea5fe0dfd51d9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:56.025252 22424 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:56.025286 22425 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:56.025337 22424 ts_tablet_manager.cc:1367] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:05:56.025349 22425 ts_tablet_manager.cc:1367] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:05:56.025782 22425 raft_consensus.cc:348] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.025882 22425 raft_consensus.cc:374] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.025911 22425 raft_consensus.cc:729] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.025866 22424 raft_consensus.cc:348] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.025952 22424 raft_consensus.cc:374] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.025978 22424 raft_consensus.cc:729] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.026039 22425 consensus_queue.cc:260] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.026072 22424 consensus_queue.cc:260] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.026314 22424 ts_tablet_manager.cc:1398] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:56.026356 22425 ts_tablet_manager.cc:1398] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:56.026479 22424 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:56.026497 22425 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:56.027319 22424 tablet_bootstrap.cc:654] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.027319 22425 tablet_bootstrap.cc:654] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.027451 22428 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:56.028255 22428 tablet_bootstrap.cc:654] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.029392 22424 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:56.029475 22424 ts_tablet_manager.cc:1367] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:05:56.029896 22424 raft_consensus.cc:348] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.030009 22424 raft_consensus.cc:374] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.030050 22424 raft_consensus.cc:729] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.030164 22424 consensus_queue.cc:260] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.030422 22424 ts_tablet_manager.cc:1398] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:56.030668 22424 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:05:56.031101 22425 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:56.031204 22425 ts_tablet_manager.cc:1367] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:05:56.031607 22425 raft_consensus.cc:348] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.031723 22425 raft_consensus.cc:374] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.031761 22425 raft_consensus.cc:729] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.031885 22425 consensus_queue.cc:260] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.032140 22425 ts_tablet_manager.cc:1398] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:56.032317 22425 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:05:56.032696 22424 tablet_bootstrap.cc:654] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.033152 22425 tablet_bootstrap.cc:654] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.034440 22425 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: No bootstrap required, opened a new log
I20240503 08:05:56.034515 22425 ts_tablet_manager.cc:1367] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:05:56.034647 22424 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee: No bootstrap required, opened a new log
I20240503 08:05:56.034706 22424 ts_tablet_manager.cc:1367] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:05:56.034957 22425 raft_consensus.cc:348] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.035049 22425 raft_consensus.cc:374] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.035080 22425 raft_consensus.cc:729] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.035106 22424 raft_consensus.cc:348] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.035185 22424 raft_consensus.cc:374] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.035214 22424 raft_consensus.cc:729] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.035188 22425 consensus_queue.cc:260] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.035427 22425 ts_tablet_manager.cc:1398] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:05:56.035933 22424 consensus_queue.cc:260] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.036175 22424 ts_tablet_manager.cc:1398] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:56.039990 22428 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:56.040102 22428 ts_tablet_manager.cc:1367] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 08:05:56.040586 22428 raft_consensus.cc:348] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.040715 22428 raft_consensus.cc:374] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.040757 22428 raft_consensus.cc:729] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.040925 22428 consensus_queue.cc:260] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.041607 22428 ts_tablet_manager.cc:1398] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:05:56.041801 22428 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:56.042789 22428 tablet_bootstrap.cc:654] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.043992 22428 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:56.044126 22428 ts_tablet_manager.cc:1367] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:05:56.044544 22428 raft_consensus.cc:348] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.044648 22428 raft_consensus.cc:374] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.044692 22428 raft_consensus.cc:729] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.044811 22428 consensus_queue.cc:260] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.045070 22428 ts_tablet_manager.cc:1398] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:56.045266 22428 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: Bootstrap starting.
I20240503 08:05:56.046167 22428 tablet_bootstrap.cc:654] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: Neither blocks nor log segments found. Creating new log.
I20240503 08:05:56.047258 22428 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: No bootstrap required, opened a new log
I20240503 08:05:56.047344 22428 ts_tablet_manager.cc:1367] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:05:56.047763 22428 raft_consensus.cc:348] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.047868 22428 raft_consensus.cc:374] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:05:56.047905 22428 raft_consensus.cc:729] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Initialized, Role: FOLLOWER
I20240503 08:05:56.048044 22428 consensus_queue.cc:260] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.048364 22428 ts_tablet_manager.cc:1398] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:56.057435 22295 raft_consensus.cc:482] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:56.057530 22295 raft_consensus.cc:504] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.057873 22295 leader_election.cc:288] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:56.058228 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d21167b87c04350b636468cdd81568f" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:56.058269 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d21167b87c04350b636468cdd81568f" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:56.058380 21816 raft_consensus.cc:2454] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:56.058398 21948 raft_consensus.cc:2454] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 0.
I20240503 08:05:56.058681 22016 leader_election.cc:302] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.058876 22295 raft_consensus.cc:2781] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:56.058945 22295 raft_consensus.cc:482] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:56.058993 22295 raft_consensus.cc:3037] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.059751 22295 raft_consensus.cc:504] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.060031 22295 leader_election.cc:288] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Requested vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:05:56.060344 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d21167b87c04350b636468cdd81568f" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:56.060344 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d21167b87c04350b636468cdd81568f" candidate_uuid: "63c7e61c2efa45ebae4113ff375dc22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:56.060457 21948 raft_consensus.cc:3037] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.060473 21816 raft_consensus.cc:3037] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.061357 21948 raft_consensus.cc:2454] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:56.061357 21816 raft_consensus.cc:2454] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c7e61c2efa45ebae4113ff375dc22c in term 1.
I20240503 08:05:56.061767 22016 leader_election.cc:302] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.061969 22295 raft_consensus.cc:2781] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:56.062088 22295 raft_consensus.cc:686] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Leader. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:05:56.062219 22295 consensus_queue.cc:237] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.063755 21667 catalog_manager.cc:5484] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } }
I20240503 08:05:56.120132 22398 raft_consensus.cc:482] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:56.120270 22398 raft_consensus.cc:504] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.121263 22398 leader_election.cc:288] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:56.124969 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe7c2f06f184762aea5fe0dfd51d9e3" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c" is_pre_election: true
I20240503 08:05:56.124971 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe7c2f06f184762aea5fe0dfd51d9e3" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee" is_pre_election: true
I20240503 08:05:56.125100 21816 raft_consensus.cc:2454] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 0.
I20240503 08:05:56.125108 22081 raft_consensus.cc:2454] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 0.
I20240503 08:05:56.125416 21883 leader_election.cc:302] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.125638 22398 raft_consensus.cc:2781] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:56.125692 22398 raft_consensus.cc:482] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:56.125730 22398 raft_consensus.cc:3037] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.126509 22398 raft_consensus.cc:504] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.126811 22398 leader_election.cc:288] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 1 election: Requested vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:56.127111 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe7c2f06f184762aea5fe0dfd51d9e3" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:05:56.127153 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe7c2f06f184762aea5fe0dfd51d9e3" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:05:56.127208 21816 raft_consensus.cc:3037] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.127250 22081 raft_consensus.cc:3037] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.127907 21816 raft_consensus.cc:2454] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 1.
I20240503 08:05:56.127965 22081 raft_consensus.cc:2454] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 1.
I20240503 08:05:56.128151 21883 leader_election.cc:302] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.128304 22398 raft_consensus.cc:2781] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:56.128392 22398 raft_consensus.cc:686] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEADER]: Becoming Leader. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Running, Role: LEADER
I20240503 08:05:56.128656 22398 consensus_queue.cc:237] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.130774 21667 catalog_manager.cc:5484] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: term changed from 0 to 1, leader changed from <none> to ddaa7bb712a245b4a37a1c511049924d (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:56.138288 22287 raft_consensus.cc:482] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:56.138386 22287 raft_consensus.cc:504] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.138754 22287 leader_election.cc:288] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:56.139101 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af6cecbb7f4422cb6c76239cf0bc002" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d" is_pre_election: true
I20240503 08:05:56.139124 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af6cecbb7f4422cb6c76239cf0bc002" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c" is_pre_election: true
I20240503 08:05:56.139219 21948 raft_consensus.cc:2454] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 0.
I20240503 08:05:56.139231 22081 raft_consensus.cc:2454] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 0.
I20240503 08:05:56.139492 21752 leader_election.cc:302] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.139659 22287 raft_consensus.cc:2781] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:05:56.139716 22287 raft_consensus.cc:482] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:05:56.139742 22287 raft_consensus.cc:3037] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.140547 22287 raft_consensus.cc:504] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.140822 22287 leader_election.cc:288] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:56.141098 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af6cecbb7f4422cb6c76239cf0bc002" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:05:56.141100 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af6cecbb7f4422cb6c76239cf0bc002" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:05:56.141194 21948 raft_consensus.cc:3037] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.141201 22081 raft_consensus.cc:3037] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:05:56.141847 21948 raft_consensus.cc:2454] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 1.
I20240503 08:05:56.141850 22081 raft_consensus.cc:2454] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 1.
I20240503 08:05:56.142098 21752 leader_election.cc:302] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:05:56.142251 22287 raft_consensus.cc:2781] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:05:56.142328 22287 raft_consensus.cc:686] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Becoming Leader. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Running, Role: LEADER
I20240503 08:05:56.142474 22287 consensus_queue.cc:237] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } }
I20240503 08:05:56.143944 21667 catalog_manager.cc:5484] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:56.261220 22441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:56.261479 22441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:56.264609 22441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:05:56.356289 21948 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.356694 22081 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.356844 21948 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.356875 21816 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.357244 22330 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:56.357345 21816 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.357673 22330 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:56.357697 22295 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:56.357950 22329 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:56.358042 22081 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:05:56.358126 22398 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:05:56.358381 22333 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.158 ms
time per row: 0.0575792 ms
I20240503 08:05:56.421360 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
/tmp/dist-test-task8ZGZ68/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:45789
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:43583
--webserver_port=34229
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:05:56.527352 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:05:56.527570 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:05:56.527631 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:05:56.530602 22476 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:05:56.530663 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:05:56.530748 22476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:05:56.534279 22476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:45789
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=34229
--tserver_master_addrs=127.0.99.190:43583
--never_fsync=true
--heap_profile_path=/tmp/kudu.22476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:05:56.535171 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:05:56.536348 22476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:05:56.543936 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:56.544240 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:05:56.544680 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:05:56.545080 22476 server_base.cc:1034] running on GCE node
I20240503 08:05:56.545406 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:05:56.545912 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:05:56.547044 22476 hybrid_clock.cc:648] HybridClock initialized: now 1714723556547025 us; error 36 us; skew 500 ppm
I20240503 08:05:56.548815 22476 webserver.cc:460] Webserver started at http://127.0.99.132:34229/ using document root <none> and password file <none>
I20240503 08:05:56.549330 22476 fs_manager.cc:359] Metadata directory not provided
I20240503 08:05:56.549396 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:05:56.552605 22476 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:05:56.554659 22490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:56.555769 22476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 08:05:56.555876 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e56f5071860c4210a63b8241a2233f00"
format_stamp: "Formatted at 2024-05-03 08:05:51 on dist-test-slave-11df"
I20240503 08:05:56.556306 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:05:56.570219 22476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:05:56.570807 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:05:56.570984 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:05:56.571512 22476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:05:56.572363 22476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:05:56.572410 22476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:56.572469 22476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:05:56.572492 22476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:05:56.580404 22476 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:45789
I20240503 08:05:56.580442 22603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:45789 every 8 connection(s)
I20240503 08:05:56.581405 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:05:56.586335 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8ZGZ68/build/debug/bin/kudu as pid 22476
I20240503 08:05:56.589887 22604 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43583
I20240503 08:05:56.590081 22604 heartbeater.cc:464] Registering TS with master...
I20240503 08:05:56.590566 22604 heartbeater.cc:511] Master 127.0.99.190:43583 requested a full tablet report, sending...
I20240503 08:05:56.591696 21675 ts_manager.cc:194] Re-registered known tserver with Master: e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789)
I20240503 08:05:56.592259 21675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45097
I20240503 08:05:56.774955 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:56.775259 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:56.775686 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:56.775897 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:56.874876 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:56.875353 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:56.876008 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:56.876292 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:56.978778 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:56.979064 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:56.980089 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:56.980607 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:57.071455 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:05:57.072126 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:05:57.072871 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:05:57.073271 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:05:57.110539 22081 consensus_queue.cc:237] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.112129 21949 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.112401 21815 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.112774 22743 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.113070 22288 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.115023 22288 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.115272 21949 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.115440 21815 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
W20240503 08:05:57.117949 22015 consensus_peers.cc:485] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d21167b87c04350b636468cdd81568f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:57.118465 21675 catalog_manager.cc:5484] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.126647 21815 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.128307 21949 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.128388 22081 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.128751 22287 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.128955 22756 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.130659 22755 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.131121 21949 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.131189 22081 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.133744 21674 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
W20240503 08:05:57.135293 21751 consensus_peers.cc:485] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 210ed86636fb4c1584cb09fc63598447. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:57.140318 22081 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.141599 21815 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.141687 21949 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.141993 22744 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:05:57.142349 22015 consensus_peers.cc:485] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e1ade447f7c4a16ae3336b3d0292233. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:57.142351 22743 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.143839 22744 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.144272 21815 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.144729 21949 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.146692 21674 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:57.154137 22081 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.155546 21815 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:05:57.155686 22015 consensus_peers.cc:485] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b3eb0eda0be496e9450d98e8a1f5b40. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:57.155754 21949 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.155910 22743 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.156128 22744 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.157754 22288 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.158179 21949 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.158289 21815 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.160534 21671 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:57.167726 22081 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.169010 21815 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:57.169023 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:05:57.169227 22015 consensus_peers.cc:485] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8260d37c2084d6a9bd2f3c6ac7e5071. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:57.169412 22743 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.169565 22288 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:57.171331 22744 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.171757 21815 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.171811 21949 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.174401 21671 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:57.594123 22604 heartbeater.cc:503] Master 127.0.99.190:43583 was elected leader, sending a full tablet report...
I20240503 08:05:57.650857 22855 ts_tablet_manager.cc:902] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:57.651631 22855 tablet_copy_client.cc:308] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:05:57.655855 22101 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:05:57.656011 22101 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 6b3eb0eda0be496e9450d98e8a1f5b40 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-6b3eb0eda0be496e9450d98e8a1f5b40
I20240503 08:05:57.657392 22101 tablet_copy_source_session.cc:205] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:57.661057 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3eb0eda0be496e9450d98e8a1f5b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:57.666280 22855 tablet_copy_client.cc:783] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:57.666450 22855 tablet_copy_client.cc:647] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:57.668188 22855 tablet_copy_client.cc:523] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:57.668552 22867 ts_tablet_manager.cc:902] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:57.669026 22867 tablet_copy_client.cc:308] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:05:57.669442 22101 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:05:57.669538 22101 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet f8260d37c2084d6a9bd2f3c6ac7e5071 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-f8260d37c2084d6a9bd2f3c6ac7e5071
I20240503 08:05:57.669919 22855 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:57.670228 22101 tablet_copy_source_session.cc:205] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:57.670993 22867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8260d37c2084d6a9bd2f3c6ac7e5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:57.673017 22867 tablet_copy_client.cc:783] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:57.673166 22867 tablet_copy_client.cc:647] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:57.674166 22867 tablet_copy_client.cc:523] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:57.675137 22867 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:57.676465 22855 log.cc:824] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Log is configured to *not* fsync() on all Append() calls
I20240503 08:05:57.688186 22874 ts_tablet_manager.cc:902] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:57.688879 22874 tablet_copy_client.cc:308] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33957
I20240503 08:05:57.692966 21837 tablet_copy_service.cc:140] P 5de0b16d88c64038b34727ac7f94d9ee: Received BeginTabletCopySession request for tablet 210ed86636fb4c1584cb09fc63598447 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:43093)
I20240503 08:05:57.693126 21837 tablet_copy_service.cc:161] P 5de0b16d88c64038b34727ac7f94d9ee: Beginning new tablet copy session on tablet 210ed86636fb4c1584cb09fc63598447 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:43093: session id = e56f5071860c4210a63b8241a2233f00-210ed86636fb4c1584cb09fc63598447
I20240503 08:05:57.694671 21837 tablet_copy_source_session.cc:205] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:57.695951 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210ed86636fb4c1584cb09fc63598447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:57.697569 22867 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:57.697605 22855 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:57.697899 22855 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:57.697889 22867 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:57.698107 22874 tablet_copy_client.cc:783] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:57.698237 22867 ts_tablet_manager.cc:1367] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240503 08:05:57.698227 22874 tablet_copy_client.cc:647] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:57.698236 22855 ts_tablet_manager.cc:1367] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20240503 08:05:57.699787 22874 tablet_copy_client.cc:523] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:57.700579 22855 raft_consensus.cc:348] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.700582 22867 raft_consensus.cc:348] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.700831 22855 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:57.700839 22867 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:57.700924 22874 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:57.701195 22855 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.701196 22867 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.702162 22867 ts_tablet_manager.cc:1398] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:05:57.702852 22101 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-f8260d37c2084d6a9bd2f3c6ac7e5071 received from {username='slave'} at 127.0.99.132:48321
I20240503 08:05:57.703037 22101 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-f8260d37c2084d6a9bd2f3c6ac7e5071 on tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:57.703054 22855 ts_tablet_manager.cc:1398] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:05:57.703514 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-6b3eb0eda0be496e9450d98e8a1f5b40 received from {username='slave'} at 127.0.99.132:48321
I20240503 08:05:57.703584 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-6b3eb0eda0be496e9450d98e8a1f5b40 on tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:57.719110 22874 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:57.719400 22874 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:57.719583 22874 ts_tablet_manager.cc:1367] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240503 08:05:57.720043 22874 raft_consensus.cc:348] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.720153 22874 raft_consensus.cc:729] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:57.720243 22874 consensus_queue.cc:260] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.720270 22867 ts_tablet_manager.cc:902] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:57.720476 22874 ts_tablet_manager.cc:1398] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:57.720782 22867 tablet_copy_client.cc:308] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:05:57.721093 21837 tablet_copy_service.cc:342] P 5de0b16d88c64038b34727ac7f94d9ee: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-210ed86636fb4c1584cb09fc63598447 received from {username='slave'} at 127.0.99.132:43093
I20240503 08:05:57.721210 21837 tablet_copy_service.cc:433] P 5de0b16d88c64038b34727ac7f94d9ee: ending tablet copy session e56f5071860c4210a63b8241a2233f00-210ed86636fb4c1584cb09fc63598447 on tablet 210ed86636fb4c1584cb09fc63598447 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:57.721251 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 2d21167b87c04350b636468cdd81568f from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:05:57.721360 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 2d21167b87c04350b636468cdd81568f from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-2d21167b87c04350b636468cdd81568f
I20240503 08:05:57.722352 22100 tablet_copy_source_session.cc:205] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:57.723196 22867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d21167b87c04350b636468cdd81568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:57.724951 22867 tablet_copy_client.cc:783] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:57.725081 22867 tablet_copy_client.cc:647] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:57.726038 22867 tablet_copy_client.cc:523] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:57.727078 22867 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:57.744971 22867 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:57.745227 22867 tablet_bootstrap.cc:492] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:57.745397 22867 ts_tablet_manager.cc:1367] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 08:05:57.745793 22867 raft_consensus.cc:348] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.745904 22867 raft_consensus.cc:729] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:57.745998 22867 consensus_queue.cc:260] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.746243 22867 ts_tablet_manager.cc:1398] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:57.746836 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-2d21167b87c04350b636468cdd81568f received from {username='slave'} at 127.0.99.132:48321
I20240503 08:05:57.746954 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-2d21167b87c04350b636468cdd81568f on tablet 2d21167b87c04350b636468cdd81568f with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:57.765491 22867 ts_tablet_manager.cc:902] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:05:57.765957 22867 tablet_copy_client.cc:308] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:05:57.766430 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 3e1ade447f7c4a16ae3336b3d0292233 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:05:57.766541 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 3e1ade447f7c4a16ae3336b3d0292233 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-3e1ade447f7c4a16ae3336b3d0292233
I20240503 08:05:57.767513 22100 tablet_copy_source_session.cc:205] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:57.768327 22867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1ade447f7c4a16ae3336b3d0292233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:57.770057 22867 tablet_copy_client.cc:783] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:57.770193 22867 tablet_copy_client.cc:647] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:57.771257 22867 tablet_copy_client.cc:523] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:57.772233 22867 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:57.788659 22867 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:57.788928 22867 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:57.789081 22867 ts_tablet_manager.cc:1367] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 08:05:57.789513 22867 raft_consensus.cc:348] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.789625 22867 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:57.789716 22867 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:57.789938 22867 ts_tablet_manager.cc:1398] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:57.790531 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-3e1ade447f7c4a16ae3336b3d0292233 received from {username='slave'} at 127.0.99.132:48321
I20240503 08:05:57.790622 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-3e1ade447f7c4a16ae3336b3d0292233 on tablet 3e1ade447f7c4a16ae3336b3d0292233 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:58.062799 22558 raft_consensus.cc:1203] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:58.146450 22558 raft_consensus.cc:1203] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:58.244279 22558 raft_consensus.cc:1203] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:58.271239 22558 raft_consensus.cc:1203] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:58.275702 22558 raft_consensus.cc:1203] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:05:58.674419 22918 raft_consensus.cc:1050] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:05:58.674965 22918 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.676337 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.676409 22558 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.676420 21815 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.676755 22997 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.676947 22918 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.677129 22743 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.679373 22997 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.679862 21949 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.680732 22558 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.682380 21674 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.683398 22944 raft_consensus.cc:1050] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:05:58.683409 21815 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.683893 22944 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.685292 22558 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.685384 22081 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.686187 22945 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.686529 22945 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.686786 21949 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.687146 23007 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.687671 22081 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.688859 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.689198 22743 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.689213 22945 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.689630 22557 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.690341 22917 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.690765 22917 raft_consensus.cc:1050] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:05:58.690749 21949 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.691121 22917 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.691280 21671 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:58.692458 21948 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.692793 22997 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.692763 22080 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.694063 21815 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.694226 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.694372 22743 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.695297 21949 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.695591 22743 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.695904 22556 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.695935 22558 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.697499 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:58.697962 22917 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.698251 21674 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.700134 22917 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.701959 21816 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.702486 22558 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.703012 21815 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.703346 21797 tablet_service.cc:1514] Processing DeleteTablet for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with delete_type TABLET_DATA_TOMBSTONED (TS 5de0b16d88c64038b34727ac7f94d9ee not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54588
I20240503 08:05:58.703995 23016 tablet_replica.cc:330] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: stopping tablet replica
I20240503 08:05:58.704452 22556 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.704622 23016 raft_consensus.cc:2229] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:58.704605 21949 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.704849 22287 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.704936 23016 raft_consensus.cc:2258] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:58.705893 21674 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.706115 23016 ts_tablet_manager.cc:1875] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:58.706300 22080 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.706657 22287 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.708851 23016 ts_tablet_manager.cc:1888] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:58.708986 23016 log.cc:1196] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8260d37c2084d6a9bd2f3c6ac7e5071
I20240503 08:05:58.709520 21660 catalog_manager.cc:4830] TS 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957): tablet f8260d37c2084d6a9bd2f3c6ac7e5071 (table rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]) successfully deleted
I20240503 08:05:58.710153 23007 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.710223 22081 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.710752 22556 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.711855 21949 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.711879 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 210ed86636fb4c1584cb09fc63598447 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:58.711969 22081 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.712201 22918 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.712657 21673 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:58.713212 22918 raft_consensus.cc:1050] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:05:58.713551 22918 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.713712 22997 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.714061 21948 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.715390 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e1ade447f7c4a16ae3336b3d0292233 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:58.715649 22558 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.715689 21815 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.716037 22917 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.716109 21948 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.716480 22917 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.716753 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.717768 23018 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.717833 22917 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.718065 21928 tablet_service.cc:1514] Processing DeleteTablet for tablet 210ed86636fb4c1584cb09fc63598447 with delete_type TABLET_DATA_TOMBSTONED (TS ddaa7bb712a245b4a37a1c511049924d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50164
I20240503 08:05:58.718437 21674 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.718752 23020 tablet_replica.cc:330] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: stopping tablet replica
I20240503 08:05:58.718765 22558 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.719856 23020 raft_consensus.cc:2229] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:58.720238 23020 raft_consensus.cc:2258] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:58.720763 21797 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1ade447f7c4a16ae3336b3d0292233 with delete_type TABLET_DATA_TOMBSTONED (TS 5de0b16d88c64038b34727ac7f94d9ee not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54588
I20240503 08:05:58.721063 22918 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.721503 23016 tablet_replica.cc:330] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: stopping tablet replica
I20240503 08:05:58.721601 23016 raft_consensus.cc:2229] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:58.721707 23016 raft_consensus.cc:2258] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:58.721737 23020 ts_tablet_manager.cc:1875] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:58.722252 23016 ts_tablet_manager.cc:1875] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:58.723152 22558 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.723898 23016 ts_tablet_manager.cc:1888] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:58.723963 23016 log.cc:1196] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3e1ade447f7c4a16ae3336b3d0292233
I20240503 08:05:58.724074 23020 ts_tablet_manager.cc:1888] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:58.724152 23020 log.cc:1196] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/210ed86636fb4c1584cb09fc63598447
I20240503 08:05:58.724737 21661 catalog_manager.cc:4830] TS ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025): tablet 210ed86636fb4c1584cb09fc63598447 (table rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]) successfully deleted
I20240503 08:05:58.724838 21660 catalog_manager.cc:4830] TS 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957): tablet 3e1ade447f7c4a16ae3336b3d0292233 (table rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]) successfully deleted
I20240503 08:05:58.725361 21673 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:58.725631 21948 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.725968 21815 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.728240 22081 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.729475 21815 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.729499 22558 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.729809 23018 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.729995 22997 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.731312 23018 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.731729 21815 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.732520 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:58.733438 21675 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:58.734093 22558 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.735916 21928 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with delete_type TABLET_DATA_TOMBSTONED (TS ddaa7bb712a245b4a37a1c511049924d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50164
I20240503 08:05:58.736150 23020 tablet_replica.cc:330] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: stopping tablet replica
I20240503 08:05:58.736308 23020 raft_consensus.cc:2229] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:58.736503 23020 raft_consensus.cc:2258] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:58.737208 23020 ts_tablet_manager.cc:1875] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:58.738813 23020 ts_tablet_manager.cc:1888] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:58.738874 23020 log.cc:1196] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6b3eb0eda0be496e9450d98e8a1f5b40
I20240503 08:05:58.739288 21661 catalog_manager.cc:4830] TS ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025): tablet 6b3eb0eda0be496e9450d98e8a1f5b40 (table rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]) successfully deleted
I20240503 08:05:58.808547 21815 consensus_queue.cc:237] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:58.809876 21948 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:05:58.810026 21751 consensus_peers.cc:485] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5af6cecbb7f4422cb6c76239cf0bc002. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:58.810242 23006 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:58.811076 22081 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:05:58.811431 23006 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:05:58.812021 22287 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:58.812429 21948 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:58.814452 21675 catalog_manager.cc:5484] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:58.816188 22081 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:05:58.822427 22917 raft_consensus.cc:1050] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:05:58.822852 22917 consensus_queue.cc:237] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.824191 21948 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.824323 22558 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.824618 22997 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.824728 21815 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:05:58.825304 22997 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.825709 22743 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:05:58.827442 22917 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.829078 21815 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.829401 21948 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.830085 21671 catalog_manager.cc:5484] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:58.832437 22558 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { replace: true } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.834677 22081 consensus_queue.cc:237] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:05:58.835851 22558 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.835909 21948 raft_consensus.cc:1261] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:05:58.836205 22918 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.836370 22997 consensus_queue.cc:1035] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:05:58.837788 22743 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.838194 22558 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.839108 21948 raft_consensus.cc:2932] T 2d21167b87c04350b636468cdd81568f P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.839466 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d21167b87c04350b636468cdd81568f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:58.840231 21673 catalog_manager.cc:5484] T 2d21167b87c04350b636468cdd81568f P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 5 to 6, VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:05:58.842695 21797 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d21167b87c04350b636468cdd81568f with delete_type TABLET_DATA_TOMBSTONED (TS 5de0b16d88c64038b34727ac7f94d9ee not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54588
I20240503 08:05:58.842964 23016 tablet_replica.cc:330] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: stopping tablet replica
I20240503 08:05:58.843088 23016 raft_consensus.cc:2229] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:05:58.843216 23016 raft_consensus.cc:2258] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:05:58.843781 23016 ts_tablet_manager.cc:1875] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:05:58.845335 23016 ts_tablet_manager.cc:1888] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:05:58.845402 23016 log.cc:1196] T 2d21167b87c04350b636468cdd81568f P 5de0b16d88c64038b34727ac7f94d9ee: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2d21167b87c04350b636468cdd81568f
I20240503 08:05:58.845808 21660 catalog_manager.cc:4830] TS 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957): tablet 2d21167b87c04350b636468cdd81568f (table rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]) successfully deleted
I20240503 08:05:59.267699 23090 ts_tablet_manager.cc:902] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:05:59.268190 23090 tablet_copy_client.cc:308] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33957
I20240503 08:05:59.268716 21837 tablet_copy_service.cc:140] P 5de0b16d88c64038b34727ac7f94d9ee: Received BeginTabletCopySession request for tablet 5af6cecbb7f4422cb6c76239cf0bc002 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:43093)
I20240503 08:05:59.268836 21837 tablet_copy_service.cc:161] P 5de0b16d88c64038b34727ac7f94d9ee: Beginning new tablet copy session on tablet 5af6cecbb7f4422cb6c76239cf0bc002 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:43093: session id = e56f5071860c4210a63b8241a2233f00-5af6cecbb7f4422cb6c76239cf0bc002
I20240503 08:05:59.269593 21837 tablet_copy_source_session.cc:205] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:05:59.270416 23090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af6cecbb7f4422cb6c76239cf0bc002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:59.272276 23090 tablet_copy_client.cc:783] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:05:59.272421 23090 tablet_copy_client.cc:647] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:59.273463 23090 tablet_copy_client.cc:523] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:59.274411 23090 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:05:59.291354 23090 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:59.291635 23090 tablet_bootstrap.cc:492] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:05:59.291793 23090 ts_tablet_manager.cc:1367] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.003s
I20240503 08:05:59.292163 23090 raft_consensus.cc:348] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:59.292270 23090 raft_consensus.cc:729] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:05:59.292359 23090 consensus_queue.cc:260] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:05:59.292605 23090 ts_tablet_manager.cc:1398] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:59.293155 21837 tablet_copy_service.cc:342] P 5de0b16d88c64038b34727ac7f94d9ee: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-5af6cecbb7f4422cb6c76239cf0bc002 received from {username='slave'} at 127.0.99.132:43093
I20240503 08:05:59.293252 21837 tablet_copy_service.cc:433] P 5de0b16d88c64038b34727ac7f94d9ee: ending tablet copy session e56f5071860c4210a63b8241a2233f00-5af6cecbb7f4422cb6c76239cf0bc002 on tablet 5af6cecbb7f4422cb6c76239cf0bc002 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:05:59.697463 22558 raft_consensus.cc:1203] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:06:00.076938 23107 raft_consensus.cc:1050] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:06:00.077397 23107 consensus_queue.cc:237] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:00.078883 21948 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:00.078908 22081 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:00.078986 22558 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:00.079396 23007 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:00.079581 23106 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:00.079800 23119 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:00.081897 23007 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.082317 22081 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.083014 22558 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.084493 21675 catalog_manager.cc:5484] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:00.084736 21948 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { replace: true } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.088147 21815 consensus_queue.cc:237] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:00.089186 22558 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:00.089221 22081 raft_consensus.cc:1261] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:00.089520 23007 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:00.089692 23106 consensus_queue.cc:1035] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:00.091080 23119 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.091528 22081 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.091563 22558 raft_consensus.cc:2932] T 5af6cecbb7f4422cb6c76239cf0bc002 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.092734 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5af6cecbb7f4422cb6c76239cf0bc002 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:06:00.093665 21671 catalog_manager.cc:5484] T 5af6cecbb7f4422cb6c76239cf0bc002 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 5 to 6, VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:00.095907 21928 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af6cecbb7f4422cb6c76239cf0bc002 with delete_type TABLET_DATA_TOMBSTONED (TS ddaa7bb712a245b4a37a1c511049924d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50164
I20240503 08:06:00.096268 23127 tablet_replica.cc:330] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: stopping tablet replica
I20240503 08:06:00.096416 23127 raft_consensus.cc:2229] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:06:00.096575 23127 raft_consensus.cc:2258] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:06:00.097220 23127 ts_tablet_manager.cc:1875] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:06:00.098903 23127 ts_tablet_manager.cc:1888] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:06:00.098992 23127 log.cc:1196] T 5af6cecbb7f4422cb6c76239cf0bc002 P ddaa7bb712a245b4a37a1c511049924d: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5af6cecbb7f4422cb6c76239cf0bc002
I20240503 08:06:00.099489 21661 catalog_manager.cc:4830] TS ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025): tablet 5af6cecbb7f4422cb6c76239cf0bc002 (table rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]) successfully deleted
I20240503 08:06:00.352361 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:00.352651 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:00.353210 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:00.353422 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:00.439302 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:00.439337 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:00.439940 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:00.441816 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:00.520716 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:00.520707 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:00.520961 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:00.521339 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:00.691167 21674 ts_manager.cc:292] Set tserver state for 63c7e61c2efa45ebae4113ff375dc22c to MAINTENANCE_MODE
I20240503 08:06:00.889537 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:00.889832 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:00.889843 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:00.891708 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:00.987432 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:00.987541 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:00.988833 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:00.995142 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:01.096800 21797 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:06:01.097558 21928 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:06:01.099141 22060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:06:01.101159 22538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:06:01.132572 22080 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:01.133863 22556 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.133867 21948 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.134240 23346 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.134479 23128 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.136217 23349 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.136574 21948 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.137853 22556 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.138679 21674 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:01.142098 22080 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.143592 21948 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:01.143685 22556 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:01.143970 23346 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:01.144146 23128 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:06:01.144287 22015 consensus_peers.cc:485] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957): Couldn't send request to peer 5de0b16d88c64038b34727ac7f94d9ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:01.146996 23128 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.147482 21948 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.147579 22556 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.149168 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:01.149919 21674 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.150053 22080 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:01.151324 21948 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.151371 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.151762 23346 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.151918 23349 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.153801 23346 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.154327 21948 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.156528 21671 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.157509 22556 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.160024 22080 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.161443 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:01.161674 21948 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:06:01.161931 22015 consensus_peers.cc:485] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957): Couldn't send request to peer 5de0b16d88c64038b34727ac7f94d9ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:01.162156 23349 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:01.162320 23346 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:01.163894 23128 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.164451 21948 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.165273 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e1ade447f7c4a16ae3336b3d0292233 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:01.165871 22556 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } } }
I20240503 08:06:01.166508 21673 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 7 to 8, NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:01.168165 22080 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:01.169381 22556 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.169479 21816 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:01.170059 23349 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.170275 23346 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:01.171980 23128 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.172384 21816 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.172547 22556 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:01.174885 21674 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:01.177897 22080 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.179116 22556 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:01.179167 21816 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:01.179530 23128 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:01.179706 23349 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:06:01.179934 22016 consensus_peers.cc:485] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025): Couldn't send request to peer ddaa7bb712a245b4a37a1c511049924d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:01.181548 23128 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:01.181952 21816 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:01.183033 22556 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:01.183198 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:01.185246 21674 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:01.188380 22080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38812
I20240503 08:06:01.188519 22080 raft_consensus.cc:595] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:01.602628 23402 ts_tablet_manager.cc:902] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:01.603351 23402 tablet_copy_client.cc:308] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:06:01.603965 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 from peer ddaa7bb712a245b4a37a1c511049924d ({username='slave'} at 127.0.99.130:33637)
I20240503 08:06:01.604110 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 6b3eb0eda0be496e9450d98e8a1f5b40 from peer ddaa7bb712a245b4a37a1c511049924d at {username='slave'} at 127.0.99.130:33637: session id = ddaa7bb712a245b4a37a1c511049924d-6b3eb0eda0be496e9450d98e8a1f5b40
I20240503 08:06:01.604950 22100 tablet_copy_source_session.cc:205] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:01.606048 23402 ts_tablet_manager.cc:1875] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:06:01.607918 23402 ts_tablet_manager.cc:1888] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:06:01.608085 23402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3eb0eda0be496e9450d98e8a1f5b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:01.609122 23402 tablet_copy_client.cc:783] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:01.609308 23402 tablet_copy_client.cc:647] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:01.609484 23346 raft_consensus.cc:979] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: : Instructing follower e56f5071860c4210a63b8241a2233f00 to start an election
I20240503 08:06:01.609570 23346 raft_consensus.cc:1067] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Signalling peer e56f5071860c4210a63b8241a2233f00 to start an election
I20240503 08:06:01.610100 22556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40"
dest_uuid: "e56f5071860c4210a63b8241a2233f00"
from {username='slave'} at 127.0.99.131:55927
I20240503 08:06:01.610312 22556 raft_consensus.cc:482] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:06:01.610478 22556 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:01.610821 23402 tablet_copy_client.cc:523] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:01.611361 22556 raft_consensus.cc:504] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.612147 23402 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:06:01.612583 22556 leader_election.cc:288] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [CANDIDATE]: Term 2 election: Requested vote from peers 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:01.613040 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "e56f5071860c4210a63b8241a2233f00" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:06:01.613157 21816 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:01.613132 22080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3eb0eda0be496e9450d98e8a1f5b40" candidate_uuid: "e56f5071860c4210a63b8241a2233f00" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:06:01.613238 22080 raft_consensus.cc:3032] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:06:01.613276 22080 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:06:01.613427 22080 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.613628 22080 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:01.613919 21816 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56f5071860c4210a63b8241a2233f00 in term 2.
I20240503 08:06:01.614531 22080 raft_consensus.cc:2454] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56f5071860c4210a63b8241a2233f00 in term 2.
I20240503 08:06:01.614936 22492 leader_election.cc:302] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c7e61c2efa45ebae4113ff375dc22c, e56f5071860c4210a63b8241a2233f00; no voters:
I20240503 08:06:01.615300 23350 raft_consensus.cc:2781] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:06:01.615463 23350 raft_consensus.cc:686] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 2 LEADER]: Becoming Leader. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Running, Role: LEADER
I20240503 08:06:01.615763 23350 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.618166 21671 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 reported cstate change: term changed from 1 to 2, leader changed from 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) to e56f5071860c4210a63b8241a2233f00 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e56f5071860c4210a63b8241a2233f00" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:01.633329 23402 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:01.633783 23402 tablet_bootstrap.cc:492] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap complete.
I20240503 08:06:01.635064 23402 ts_tablet_manager.cc:1367] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.003s
I20240503 08:06:01.636482 23402 raft_consensus.cc:348] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.636746 23402 raft_consensus.cc:729] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: LEARNER
I20240503 08:06:01.636898 23402 consensus_queue.cc:260] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:01.637264 23402 ts_tablet_manager.cc:1398] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:06:01.637929 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session ddaa7bb712a245b4a37a1c511049924d-6b3eb0eda0be496e9450d98e8a1f5b40 received from {username='slave'} at 127.0.99.130:33637
I20240503 08:06:01.638016 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session ddaa7bb712a245b4a37a1c511049924d-6b3eb0eda0be496e9450d98e8a1f5b40 on tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with peer ddaa7bb712a245b4a37a1c511049924d
I20240503 08:06:01.676329 23425 ts_tablet_manager.cc:902] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:01.677058 23425 tablet_copy_client.cc:308] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:06:01.677637 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 from peer 5de0b16d88c64038b34727ac7f94d9ee ({username='slave'} at 127.0.99.129:53743)
I20240503 08:06:01.677754 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet f8260d37c2084d6a9bd2f3c6ac7e5071 from peer 5de0b16d88c64038b34727ac7f94d9ee at {username='slave'} at 127.0.99.129:53743: session id = 5de0b16d88c64038b34727ac7f94d9ee-f8260d37c2084d6a9bd2f3c6ac7e5071
I20240503 08:06:01.678462 22100 tablet_copy_source_session.cc:205] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:01.679430 23425 ts_tablet_manager.cc:1875] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:06:01.681161 23425 ts_tablet_manager.cc:1888] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:06:01.681358 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8260d37c2084d6a9bd2f3c6ac7e5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:01.682317 23425 tablet_copy_client.cc:783] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:01.682451 23425 tablet_copy_client.cc:647] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:01.683969 23425 tablet_copy_client.cc:523] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:01.685243 23425 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:06:01.704493 23425 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:01.704802 23425 tablet_bootstrap.cc:492] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap complete.
I20240503 08:06:01.704979 23425 ts_tablet_manager.cc:1367] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240503 08:06:01.705390 23425 raft_consensus.cc:348] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.705503 23425 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: LEARNER
I20240503 08:06:01.705588 23425 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.705860 23425 ts_tablet_manager.cc:1398] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:06:01.706549 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session 5de0b16d88c64038b34727ac7f94d9ee-f8260d37c2084d6a9bd2f3c6ac7e5071 received from {username='slave'} at 127.0.99.129:53743
I20240503 08:06:01.706704 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session 5de0b16d88c64038b34727ac7f94d9ee-f8260d37c2084d6a9bd2f3c6ac7e5071 on tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with peer 5de0b16d88c64038b34727ac7f94d9ee
I20240503 08:06:01.781749 23425 ts_tablet_manager.cc:902] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:01.782296 23425 tablet_copy_client.cc:308] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:06:01.782855 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 3e1ade447f7c4a16ae3336b3d0292233 from peer 5de0b16d88c64038b34727ac7f94d9ee ({username='slave'} at 127.0.99.129:53743)
I20240503 08:06:01.783002 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 3e1ade447f7c4a16ae3336b3d0292233 from peer 5de0b16d88c64038b34727ac7f94d9ee at {username='slave'} at 127.0.99.129:53743: session id = 5de0b16d88c64038b34727ac7f94d9ee-3e1ade447f7c4a16ae3336b3d0292233
I20240503 08:06:01.783747 22100 tablet_copy_source_session.cc:205] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:01.784421 23425 ts_tablet_manager.cc:1875] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:06:01.786176 23425 ts_tablet_manager.cc:1888] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:06:01.786321 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1ade447f7c4a16ae3336b3d0292233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:01.787178 23425 tablet_copy_client.cc:783] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:01.787302 23425 tablet_copy_client.cc:647] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:01.788410 23425 tablet_copy_client.cc:523] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:01.789364 23425 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap starting.
I20240503 08:06:01.806845 23425 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:01.807173 23425 tablet_bootstrap.cc:492] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Bootstrap complete.
I20240503 08:06:01.807348 23425 ts_tablet_manager.cc:1367] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 08:06:01.807790 23425 raft_consensus.cc:348] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.807919 23425 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Initialized, Role: LEARNER
I20240503 08:06:01.808023 23425 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: true } }
I20240503 08:06:01.808290 23425 ts_tablet_manager.cc:1398] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:06:01.808899 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session 5de0b16d88c64038b34727ac7f94d9ee-3e1ade447f7c4a16ae3336b3d0292233 received from {username='slave'} at 127.0.99.129:53743
I20240503 08:06:01.808979 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session 5de0b16d88c64038b34727ac7f94d9ee-3e1ade447f7c4a16ae3336b3d0292233 on tablet 3e1ade447f7c4a16ae3336b3d0292233 with peer 5de0b16d88c64038b34727ac7f94d9ee
I20240503 08:06:02.008224 21816 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:06:02.008714 23350 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.011502 22080 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:06:02.012045 23350 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.016705 21949 raft_consensus.cc:3037] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Advancing to term 2
I20240503 08:06:02.017585 21949 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 2 LEARNER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:06:02.017946 23452 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.019819 23350 raft_consensus.cc:1050] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00: attempting to promote NON_VOTER ddaa7bb712a245b4a37a1c511049924d to VOTER
I20240503 08:06:02.020531 23350 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } }
I20240503 08:06:02.021853 21816 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:02.021863 22080 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:02.022269 23459 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:02.022424 21949 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 2 LEARNER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:02.022467 23407 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:02.022754 23350 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:02.024163 23407 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.024526 21949 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.025847 22080 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.026414 21671 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 8 to 10, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e56f5071860c4210a63b8241a2233f00" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:02.028352 21816 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.034090 22556 consensus_queue.cc:237] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } }
I20240503 08:06:02.035135 21816 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:02.035126 21949 raft_consensus.cc:1261] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Refusing update from remote peer e56f5071860c4210a63b8241a2233f00: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:02.035562 23452 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:02.035737 23350 consensus_queue.cc:1035] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:02.037148 23350 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.037658 21816 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.038564 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:06:02.039254 21673 catalog_manager.cc:5484] T 6b3eb0eda0be496e9450d98e8a1f5b40 P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 10 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e56f5071860c4210a63b8241a2233f00" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:02.040073 21949 raft_consensus.cc:2932] T 6b3eb0eda0be496e9450d98e8a1f5b40 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:02.041761 22060 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3eb0eda0be496e9450d98e8a1f5b40 with delete_type TABLET_DATA_TOMBSTONED (TS 63c7e61c2efa45ebae4113ff375dc22c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38256
I20240503 08:06:02.042488 23461 tablet_replica.cc:330] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: stopping tablet replica
I20240503 08:06:02.042826 23461 raft_consensus.cc:2229] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:06:02.043044 23461 raft_consensus.cc:2258] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:06:02.044085 23461 ts_tablet_manager.cc:1875] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:06:02.045814 23461 ts_tablet_manager.cc:1888] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:06:02.045888 23461 log.cc:1196] T 6b3eb0eda0be496e9450d98e8a1f5b40 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b3eb0eda0be496e9450d98e8a1f5b40
I20240503 08:06:02.046387 21660 catalog_manager.cc:4830] TS 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301): tablet 6b3eb0eda0be496e9450d98e8a1f5b40 (table rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]) successfully deleted
I20240503 08:06:02.208834 21816 raft_consensus.cc:1203] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:06:02.235532 21816 raft_consensus.cc:1203] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:06:02.290485 21949 consensus_queue.cc:237] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } }
I20240503 08:06:02.291782 22080 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:02.291828 21816 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:02.292181 23485 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:02.292390 22475 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:02.294095 23488 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:02.294442 21816 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:02.296617 21675 catalog_manager.cc:5484] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 08:06:02.296900 22080 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:02.299902 21949 consensus_queue.cc:237] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:02.302167 21816 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:02.302644 23488 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:02.303135 22080 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:02.303663 23488 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:02.304492 22475 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:02.306280 21661 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:02.306272 21816 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
W20240503 08:06:02.306777 21883 consensus_peers.cc:485] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebe7c2f06f184762aea5fe0dfd51d9e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:02.307251 22081 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:02.308112 21675 catalog_manager.cc:5484] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:02.655603 23478 raft_consensus.cc:1050] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee to VOTER
I20240503 08:06:02.656085 23478 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:02.657513 21816 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.657547 22556 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.657548 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.657989 23521 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.658201 23478 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.658381 23477 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.660355 23477 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.660787 22556 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.661242 21816 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.663172 21949 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.663318 21671 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:02.745738 22081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48560
I20240503 08:06:02.745887 22081 raft_consensus.cc:595] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:02.778813 23545 ts_tablet_manager.cc:902] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025)
I20240503 08:06:02.779300 23545 tablet_copy_client.cc:308] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44025
I20240503 08:06:02.779894 21969 tablet_copy_service.cc:140] P ddaa7bb712a245b4a37a1c511049924d: Received BeginTabletCopySession request for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:39639)
I20240503 08:06:02.780038 21969 tablet_copy_service.cc:161] P ddaa7bb712a245b4a37a1c511049924d: Beginning new tablet copy session on tablet ebe7c2f06f184762aea5fe0dfd51d9e3 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:39639: session id = e56f5071860c4210a63b8241a2233f00-ebe7c2f06f184762aea5fe0dfd51d9e3
I20240503 08:06:02.781230 21969 tablet_copy_source_session.cc:205] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:02.782344 23545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe7c2f06f184762aea5fe0dfd51d9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:02.784166 23545 tablet_copy_client.cc:783] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:02.784303 23545 tablet_copy_client.cc:647] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:02.785568 23545 tablet_copy_client.cc:523] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:02.786528 23545 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:06:02.803814 23545 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:02.804090 23545 tablet_bootstrap.cc:492] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:06:02.804248 23545 ts_tablet_manager.cc:1367] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.003s
I20240503 08:06:02.804642 23545 raft_consensus.cc:348] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:02.804765 23545 raft_consensus.cc:729] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:06:02.804872 23545 consensus_queue.cc:260] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:02.805143 23545 ts_tablet_manager.cc:1398] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:06:02.805812 21969 tablet_copy_service.cc:342] P ddaa7bb712a245b4a37a1c511049924d: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-ebe7c2f06f184762aea5fe0dfd51d9e3 received from {username='slave'} at 127.0.99.132:39639
I20240503 08:06:02.805908 21969 tablet_copy_service.cc:433] P ddaa7bb712a245b4a37a1c511049924d: ending tablet copy session e56f5071860c4210a63b8241a2233f00-ebe7c2f06f184762aea5fe0dfd51d9e3 on tablet ebe7c2f06f184762aea5fe0dfd51d9e3 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:06:02.817293 23349 raft_consensus.cc:1050] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER 5de0b16d88c64038b34727ac7f94d9ee to VOTER
I20240503 08:06:02.817736 23349 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:02.819079 21816 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.819079 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.819114 21949 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:02.819543 23521 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.819765 23477 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.819924 23349 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:02.821614 23349 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.822166 21949 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.822244 22556 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.823300 21816 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.824910 21674 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 8 to 9, 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:02.969805 22081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48570
I20240503 08:06:02.969940 22081 raft_consensus.cc:595] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:02.984644 22081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48582
I20240503 08:06:02.984786 22081 raft_consensus.cc:595] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:03.133001 23478 raft_consensus.cc:979] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: : Instructing follower 5de0b16d88c64038b34727ac7f94d9ee to start an election
I20240503 08:06:03.133126 23478 raft_consensus.cc:1067] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Signalling peer 5de0b16d88c64038b34727ac7f94d9ee to start an election
I20240503 08:06:03.133579 21816 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071"
dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
from {username='slave'} at 127.0.99.131:53119
I20240503 08:06:03.133738 21816 raft_consensus.cc:482] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:06:03.133801 21816 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.134621 21816 raft_consensus.cc:504] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.135097 21816 leader_election.cc:288] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 2 election: Requested vote from peers ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025), 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301), e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789)
I20240503 08:06:03.135417 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:06:03.135411 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
I20240503 08:06:03.135541 21949 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.135541 22081 raft_consensus.cc:3032] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:06:03.135587 22081 raft_consensus.cc:729] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:06:03.135592 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8260d37c2084d6a9bd2f3c6ac7e5071" candidate_uuid: "5de0b16d88c64038b34727ac7f94d9ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e56f5071860c4210a63b8241a2233f00"
I20240503 08:06:03.135769 22556 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.135725 22081 consensus_queue.cc:260] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.135955 22081 raft_consensus.cc:3037] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.136547 22556 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 2.
I20240503 08:06:03.136569 21949 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 2.
I20240503 08:06:03.136858 22081 raft_consensus.cc:2454] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de0b16d88c64038b34727ac7f94d9ee in term 2.
I20240503 08:06:03.136991 21751 leader_election.cc:302] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, ddaa7bb712a245b4a37a1c511049924d, e56f5071860c4210a63b8241a2233f00; no voters:
I20240503 08:06:03.137193 23555 raft_consensus.cc:2781] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:06:03.137274 23555 raft_consensus.cc:686] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 LEADER]: Becoming Leader. State: Replica: 5de0b16d88c64038b34727ac7f94d9ee, State: Running, Role: LEADER
I20240503 08:06:03.137465 23555 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.139176 21671 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: term changed from 1 to 2, leader changed from 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) to 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:03.207327 22081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48594
I20240503 08:06:03.207466 22081 raft_consensus.cc:595] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:03.225093 22556 raft_consensus.cc:1203] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:06:03.226577 23349 raft_consensus.cc:979] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: : Instructing follower ddaa7bb712a245b4a37a1c511049924d to start an election
I20240503 08:06:03.226696 23349 raft_consensus.cc:1067] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Signalling peer ddaa7bb712a245b4a37a1c511049924d to start an election
I20240503 08:06:03.227210 21949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233"
dest_uuid: "ddaa7bb712a245b4a37a1c511049924d"
from {username='slave'} at 127.0.99.131:54499
I20240503 08:06:03.227377 21949 raft_consensus.cc:482] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:06:03.227442 21949 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.228345 21949 raft_consensus.cc:504] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.228734 21949 leader_election.cc:288] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 2 election: Requested vote from peers 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301), e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789), 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:06:03.229108 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
I20240503 08:06:03.229216 22081 raft_consensus.cc:3032] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:06:03.229262 22081 raft_consensus.cc:729] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c7e61c2efa45ebae4113ff375dc22c, State: Running, Role: LEADER
I20240503 08:06:03.229249 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "5de0b16d88c64038b34727ac7f94d9ee"
I20240503 08:06:03.229382 21816 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.229363 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1ade447f7c4a16ae3336b3d0292233" candidate_uuid: "ddaa7bb712a245b4a37a1c511049924d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e56f5071860c4210a63b8241a2233f00"
I20240503 08:06:03.229396 22081 consensus_queue.cc:260] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.229465 22556 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.229610 22081 raft_consensus.cc:3037] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:06:03.230389 21816 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 2.
I20240503 08:06:03.230393 22556 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 2.
I20240503 08:06:03.230432 22081 raft_consensus.cc:2454] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa7bb712a245b4a37a1c511049924d in term 2.
I20240503 08:06:03.230719 21883 leader_election.cc:302] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5de0b16d88c64038b34727ac7f94d9ee, 63c7e61c2efa45ebae4113ff375dc22c, ddaa7bb712a245b4a37a1c511049924d; no voters:
I20240503 08:06:03.230901 23603 raft_consensus.cc:2781] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:06:03.231015 23603 raft_consensus.cc:686] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 2 LEADER]: Becoming Leader. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Running, Role: LEADER
I20240503 08:06:03.231154 23603 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.232664 21675 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: term changed from 1 to 2, leader changed from 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) to ddaa7bb712a245b4a37a1c511049924d (127.0.99.130). New cstate: current_term: 2 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:03.666275 22556 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.666707 23555 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.676450 21816 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.676887 23603 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.716976 22081 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.717481 23555 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.721045 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.721459 23636 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.769577 23643 raft_consensus.cc:1050] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:06:03.769975 23643 consensus_queue.cc:237] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:03.771246 22556 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:03.771340 21816 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:03.771401 22081 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:03.771662 23645 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:03.771842 23643 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:03.771987 23603 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:03.773886 23645 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.774374 22081 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.774492 22556 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.776424 21816 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.776701 21671 catalog_manager.cc:5484] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:03.780134 21949 consensus_queue.cc:237] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:03.781386 21816 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:03.781405 22556 raft_consensus.cc:1261] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:03.781759 23643 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:03.782058 23603 consensus_queue.cc:1035] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:03.782580 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.783146 23645 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.783543 23646 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.783959 21816 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.783959 22556 raft_consensus.cc:2932] T ebe7c2f06f184762aea5fe0dfd51d9e3 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:03.785328 21661 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:06:03.785744 22081 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:06:03.786437 23603 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:06:03.786885 21673 catalog_manager.cc:5484] T ebe7c2f06f184762aea5fe0dfd51d9e3 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 6 to 7, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:03.789701 22060 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe7c2f06f184762aea5fe0dfd51d9e3 with delete_type TABLET_DATA_TOMBSTONED (TS 63c7e61c2efa45ebae4113ff375dc22c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38256
I20240503 08:06:03.790011 23655 tablet_replica.cc:330] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: stopping tablet replica
I20240503 08:06:03.790077 21949 consensus_queue.cc:237] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:03.790163 23655 raft_consensus.cc:2229] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:06:03.790326 23655 raft_consensus.cc:2258] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:06:03.791208 22556 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:03.791463 23655 ts_tablet_manager.cc:1875] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:06:03.791493 21816 raft_consensus.cc:1261] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Refusing update from remote peer ddaa7bb712a245b4a37a1c511049924d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:03.791556 23646 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:03.792102 23646 consensus_queue.cc:1035] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:03.792968 23603 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:03.793380 22556 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:03.793781 21816 raft_consensus.cc:2932] T 3e1ade447f7c4a16ae3336b3d0292233 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:03.794631 21661 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e1ade447f7c4a16ae3336b3d0292233 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:06:03.794776 23655 ts_tablet_manager.cc:1888] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:06:03.794850 23655 log.cc:1196] T ebe7c2f06f184762aea5fe0dfd51d9e3 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ebe7c2f06f184762aea5fe0dfd51d9e3
I20240503 08:06:03.795359 21660 catalog_manager.cc:4830] TS 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301): tablet ebe7c2f06f184762aea5fe0dfd51d9e3 (table rebalance_test_table_2 [id=ea4c9577ad1b441cbc93cf021d3ce33b]) successfully deleted
I20240503 08:06:03.795423 21671 catalog_manager.cc:5484] T 3e1ade447f7c4a16ae3336b3d0292233 P ddaa7bb712a245b4a37a1c511049924d reported cstate change: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "ddaa7bb712a245b4a37a1c511049924d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:03.797493 22060 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1ade447f7c4a16ae3336b3d0292233 with delete_type TABLET_DATA_TOMBSTONED (TS 63c7e61c2efa45ebae4113ff375dc22c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38256
I20240503 08:06:03.797680 23655 tablet_replica.cc:330] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: stopping tablet replica
I20240503 08:06:03.797811 23655 raft_consensus.cc:2229] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:06:03.797902 23655 raft_consensus.cc:2258] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:06:03.798439 23655 ts_tablet_manager.cc:1875] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:06:03.799816 23655 ts_tablet_manager.cc:1888] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:06:03.799873 23655 log.cc:1196] T 3e1ade447f7c4a16ae3336b3d0292233 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3e1ade447f7c4a16ae3336b3d0292233
I20240503 08:06:03.800251 21660 catalog_manager.cc:4830] TS 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301): tablet 3e1ade447f7c4a16ae3336b3d0292233 (table rebalance_test_table_0 [id=ba451309454c49caa7df888519db4b19]) successfully deleted
I20240503 08:06:04.065322 21816 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:04.066571 22081 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:04.066637 22556 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:06:04.067003 23638 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:04.067184 23636 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:06:04.068811 23555 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:04.069279 22081 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:04.069371 22556 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:04.071183 21673 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:04.074309 21816 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:04.075484 22556 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:06:04.075901 23555 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:04.076042 22081 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:06:04.076087 21752 consensus_peers.cc:485] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee -> Peer ddaa7bb712a245b4a37a1c511049924d (127.0.99.130:44025): Couldn't send request to peer ddaa7bb712a245b4a37a1c511049924d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:04.076468 23555 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:06:04.077776 23638 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:04.078316 22556 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:04.079325 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 210ed86636fb4c1584cb09fc63598447 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:04.080088 22081 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } } }
I20240503 08:06:04.080492 21675 catalog_manager.cc:5484] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 7 to 8, NON_VOTER ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:04.083276 22081 consensus_queue.cc:237] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } }
I20240503 08:06:04.084378 21949 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:04.084450 21816 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:04.084731 23654 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:04.084873 23349 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:04.086550 23349 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:04.087018 21949 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:04.087348 21816 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } }
I20240503 08:06:04.088759 21673 catalog_manager.cc:5484] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 08:06:04.091924 22081 consensus_queue.cc:237] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:04.093063 21949 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:06:04.093262 22015 consensus_peers.cc:485] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59ddd92adecf4d75a6cd209cb59788fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:04.093495 21816 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:04.093796 23349 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:04.093974 23693 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:04.095768 23654 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.096238 21816 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.098132 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59ddd92adecf4d75a6cd209cb59788fc with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:04.098443 21674 catalog_manager.cc:5484] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:06:04.098839 21949 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.288157 21816 consensus_queue.cc:237] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } }
I20240503 08:06:04.289316 21949 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:04.289338 22556 raft_consensus.cc:1261] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:06:04.289661 23636 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:04.289847 23555 consensus_queue.cc:1035] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:06:04.291352 23638 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 5de0b16d88c64038b34727ac7f94d9ee [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:04.291808 22556 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:04.291792 21949 raft_consensus.cc:2932] T f8260d37c2084d6a9bd2f3c6ac7e5071 P ddaa7bb712a245b4a37a1c511049924d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:04.293572 21660 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:06:04.294127 21673 catalog_manager.cc:5484] T f8260d37c2084d6a9bd2f3c6ac7e5071 P e56f5071860c4210a63b8241a2233f00 reported cstate change: config changed from index 9 to 11, VOTER 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "5de0b16d88c64038b34727ac7f94d9ee" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } attrs { promote: false } } }
I20240503 08:06:04.296392 22060 tablet_service.cc:1514] Processing DeleteTablet for tablet f8260d37c2084d6a9bd2f3c6ac7e5071 with delete_type TABLET_DATA_TOMBSTONED (TS 63c7e61c2efa45ebae4113ff375dc22c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38256
I20240503 08:06:04.296612 23655 tablet_replica.cc:330] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: stopping tablet replica
I20240503 08:06:04.296742 23655 raft_consensus.cc:2229] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:06:04.296859 23655 raft_consensus.cc:2258] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:06:04.297467 23655 ts_tablet_manager.cc:1875] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:06:04.299229 23655 ts_tablet_manager.cc:1888] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:06:04.299304 23655 log.cc:1196] T f8260d37c2084d6a9bd2f3c6ac7e5071 P 63c7e61c2efa45ebae4113ff375dc22c: Deleting WAL directory at /tmp/dist-test-task8ZGZ68/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714723107332537-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8260d37c2084d6a9bd2f3c6ac7e5071
I20240503 08:06:04.299767 21660 catalog_manager.cc:4830] TS 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301): tablet f8260d37c2084d6a9bd2f3c6ac7e5071 (table rebalance_test_table_1 [id=4c54b281c475482b90522942c8182bcb]) successfully deleted
I20240503 08:06:04.485189 23721 ts_tablet_manager.cc:902] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:04.485697 23721 tablet_copy_client.cc:308] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:06:04.486205 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet 59ddd92adecf4d75a6cd209cb59788fc from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:06:04.486320 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet 59ddd92adecf4d75a6cd209cb59788fc from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-59ddd92adecf4d75a6cd209cb59788fc
I20240503 08:06:04.487102 22100 tablet_copy_source_session.cc:205] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:04.487979 23721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ddd92adecf4d75a6cd209cb59788fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:04.489789 23721 tablet_copy_client.cc:783] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:04.489928 23721 tablet_copy_client.cc:647] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:04.491379 23721 tablet_copy_client.cc:523] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:04.492388 23721 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:06:04.509936 23721 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:04.510198 23721 tablet_bootstrap.cc:492] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:06:04.510340 23721 ts_tablet_manager.cc:1367] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 08:06:04.510754 23721 raft_consensus.cc:348] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:04.510893 23721 raft_consensus.cc:729] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:06:04.511019 23721 consensus_queue.cc:260] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:04.511301 23721 ts_tablet_manager.cc:1398] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:06:04.511770 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-59ddd92adecf4d75a6cd209cb59788fc received from {username='slave'} at 127.0.99.132:48321
I20240503 08:06:04.511857 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-59ddd92adecf4d75a6cd209cb59788fc on tablet 59ddd92adecf4d75a6cd209cb59788fc with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:06:04.533573 22081 consensus_queue.cc:237] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } }
I20240503 08:06:04.534878 21949 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:04.534899 21816 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:06:04.535349 23695 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:04.535524 23693 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:06:04.537214 23654 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } }
I20240503 08:06:04.537631 21816 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } }
I20240503 08:06:04.538985 21949 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } }
I20240503 08:06:04.539497 21671 catalog_manager.cc:5484] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } }
I20240503 08:06:04.542827 22081 consensus_queue.cc:237] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:04.543996 21949 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:06:04.544293 21816 raft_consensus.cc:1261] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:06:04.544898 22015 consensus_peers.cc:485] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c -> Peer e56f5071860c4210a63b8241a2233f00 (127.0.99.132:45789): Couldn't send request to peer e56f5071860c4210a63b8241a2233f00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b54dfa529ecc4aa2ae4056ed6c8fe2c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:06:04.544927 23654 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:04.545118 23695 consensus_queue.cc:1035] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:06:04.546695 23695 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.547048 21816 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.548720 21660 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:06:04.549314 21673 catalog_manager.cc:5484] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 5de0b16d88c64038b34727ac7f94d9ee reported cstate change: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.549436 21948 raft_consensus.cc:2932] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e56f5071860c4210a63b8241a2233f00 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } } }
I20240503 08:06:04.609813 23750 ts_tablet_manager.cc:902] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Initiating tablet copy from peer 5de0b16d88c64038b34727ac7f94d9ee (127.0.99.129:33957)
I20240503 08:06:04.610316 23750 tablet_copy_client.cc:308] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33957
I20240503 08:06:04.610783 21837 tablet_copy_service.cc:140] P 5de0b16d88c64038b34727ac7f94d9ee: Received BeginTabletCopySession request for tablet 210ed86636fb4c1584cb09fc63598447 from peer ddaa7bb712a245b4a37a1c511049924d ({username='slave'} at 127.0.99.130:47901)
I20240503 08:06:04.610883 21837 tablet_copy_service.cc:161] P 5de0b16d88c64038b34727ac7f94d9ee: Beginning new tablet copy session on tablet 210ed86636fb4c1584cb09fc63598447 from peer ddaa7bb712a245b4a37a1c511049924d at {username='slave'} at 127.0.99.130:47901: session id = ddaa7bb712a245b4a37a1c511049924d-210ed86636fb4c1584cb09fc63598447
I20240503 08:06:04.611651 21837 tablet_copy_source_session.cc:205] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:04.612291 23750 ts_tablet_manager.cc:1875] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:06:04.614069 23750 ts_tablet_manager.cc:1888] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:06:04.614210 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 210ed86636fb4c1584cb09fc63598447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:04.615134 23750 tablet_copy_client.cc:783] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:04.615267 23750 tablet_copy_client.cc:647] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:04.616343 23750 tablet_copy_client.cc:523] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:04.617342 23750 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap starting.
I20240503 08:06:04.637446 23750 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:04.638288 23750 tablet_bootstrap.cc:492] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Bootstrap complete.
I20240503 08:06:04.638473 23750 ts_tablet_manager.cc:1367] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240503 08:06:04.638954 23750 raft_consensus.cc:348] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:04.639135 23750 raft_consensus.cc:729] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ddaa7bb712a245b4a37a1c511049924d, State: Initialized, Role: LEARNER
I20240503 08:06:04.639235 23750 consensus_queue.cc:260] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: true } }
I20240503 08:06:04.639495 23750 ts_tablet_manager.cc:1398] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:06:04.640071 21837 tablet_copy_service.cc:342] P 5de0b16d88c64038b34727ac7f94d9ee: Request end of tablet copy session ddaa7bb712a245b4a37a1c511049924d-210ed86636fb4c1584cb09fc63598447 received from {username='slave'} at 127.0.99.130:47901
I20240503 08:06:04.640165 21837 tablet_copy_service.cc:433] P 5de0b16d88c64038b34727ac7f94d9ee: ending tablet copy session ddaa7bb712a245b4a37a1c511049924d-210ed86636fb4c1584cb09fc63598447 on tablet 210ed86636fb4c1584cb09fc63598447 with peer ddaa7bb712a245b4a37a1c511049924d
I20240503 08:06:04.892833 22556 raft_consensus.cc:1203] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:06:04.997648 21948 raft_consensus.cc:1203] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:06:05.015851 23787 ts_tablet_manager.cc:902] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Initiating tablet copy from peer 63c7e61c2efa45ebae4113ff375dc22c (127.0.99.131:35301)
I20240503 08:06:05.016319 23787 tablet_copy_client.cc:308] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:35301
I20240503 08:06:05.016811 22100 tablet_copy_service.cc:140] P 63c7e61c2efa45ebae4113ff375dc22c: Received BeginTabletCopySession request for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 from peer e56f5071860c4210a63b8241a2233f00 ({username='slave'} at 127.0.99.132:48321)
I20240503 08:06:05.016916 22100 tablet_copy_service.cc:161] P 63c7e61c2efa45ebae4113ff375dc22c: Beginning new tablet copy session on tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 from peer e56f5071860c4210a63b8241a2233f00 at {username='slave'} at 127.0.99.132:48321: session id = e56f5071860c4210a63b8241a2233f00-b54dfa529ecc4aa2ae4056ed6c8fe2c3
I20240503 08:06:05.017697 22100 tablet_copy_source_session.cc:205] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P 63c7e61c2efa45ebae4113ff375dc22c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:06:05.018504 23787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:06:05.020282 23787 tablet_copy_client.cc:783] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 0 data blocks...
I20240503 08:06:05.020414 23787 tablet_copy_client.cc:647] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:06:05.022336 23787 tablet_copy_client.cc:523] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:06:05.023367 23787 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Bootstrap starting.
I20240503 08:06:05.040953 23787 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:06:05.041245 23787 tablet_bootstrap.cc:492] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Bootstrap complete.
I20240503 08:06:05.041442 23787 ts_tablet_manager.cc:1367] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240503 08:06:05.041903 23787 raft_consensus.cc:348] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:05.042038 23787 raft_consensus.cc:729] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e56f5071860c4210a63b8241a2233f00, State: Initialized, Role: LEARNER
I20240503 08:06:05.042145 23787 consensus_queue.cc:260] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: true } }
I20240503 08:06:05.042426 23787 ts_tablet_manager.cc:1398] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:06:05.042903 22100 tablet_copy_service.cc:342] P 63c7e61c2efa45ebae4113ff375dc22c: Request end of tablet copy session e56f5071860c4210a63b8241a2233f00-b54dfa529ecc4aa2ae4056ed6c8fe2c3 received from {username='slave'} at 127.0.99.132:48321
I20240503 08:06:05.043017 22100 tablet_copy_service.cc:433] P 63c7e61c2efa45ebae4113ff375dc22c: ending tablet copy session e56f5071860c4210a63b8241a2233f00-b54dfa529ecc4aa2ae4056ed6c8fe2c3 on tablet b54dfa529ecc4aa2ae4056ed6c8fe2c3 with peer e56f5071860c4210a63b8241a2233f00
I20240503 08:06:05.367853 23654 raft_consensus.cc:1050] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c: attempting to promote NON_VOTER e56f5071860c4210a63b8241a2233f00 to VOTER
I20240503 08:06:05.368247 23654 consensus_queue.cc:237] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } }
I20240503 08:06:05.369560 22556 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:05.369582 21948 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:05.369661 21816 raft_consensus.cc:1261] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Refusing update from remote peer 63c7e61c2efa45ebae4113ff375dc22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:06:05.369961 23809 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:05.370419 23693 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:05.370667 23694 consensus_queue.cc:1035] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:06:05.372531 23693 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:05.373036 21816 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P 5de0b16d88c64038b34727ac7f94d9ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:05.373000 21948 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:05.375213 21673 catalog_manager.cc:5484] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c reported cstate change: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c7e61c2efa45ebae4113ff375dc22c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:06:05.376353 22556 raft_consensus.cc:2932] T 59ddd92adecf4d75a6cd209cb59788fc P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e56f5071860c4210a63b8241a2233f00 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } }
I20240503 08:06:05.441293 22081 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59ddd92adecf4d75a6cd209cb59788fc"
dest_uuid: "63c7e61c2efa45ebae4113ff375dc22c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48736
I20240503 08:06:05.441463 22081 raft_consensus.cc:595] T 59ddd92adecf4d75a6cd209cb59788fc P 63c7e61c2efa45ebae4113ff375dc22c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:06:05.458014 22556 raft_consensus.cc:1203] T b54dfa529ecc4aa2ae4056ed6c8fe2c3 P e56f5071860c4210a63b8241a2233f00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:06:05.554013 23836 raft_consensus.cc:1050] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee: attempting to promote NON_VOTER ddaa7bb712a245b4a37a1c511049924d to VOTER
I20240503 08:06:05.554435 23836 consensus_queue.cc:237] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } }
I20240503 08:06:05.555824 22556 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P e56f5071860c4210a63b8241a2233f00 [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:05.555858 22081 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P 63c7e61c2efa45ebae4113ff375dc22c [term 1 FOLLOWER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:05.556201 21948 raft_consensus.cc:1261] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 LEARNER]: Refusing update from remote peer 5de0b16d88c64038b34727ac7f94d9ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:06:05.556217 23839 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:05.556445 23838 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:05.556597 23836 consensus_queue.cc:1035] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:06:05.558468 23839 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P 5de0b16d88c64038b34727ac7f94d9ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 45789 } attrs { promote: false } } peers { permanent_uuid: "ddaa7bb712a245b4a37a1c511049924d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44025 } attrs { promote: false } } }
I20240503 08:06:05.558872 21948 raft_consensus.cc:2932] T 210ed86636fb4c1584cb09fc63598447 P ddaa7bb712a245b4a37a1c511049924d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ddaa7bb712a245b4a37a1c511049924d (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5de0b16d88c64038b34727ac7f94d9ee" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33957 } } peers { permanent_uuid: "63c7e61c2efa45ebae4113ff375dc22c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35301 } attrs { replace: true } } peers { permanent_uuid: "e56f5071860c4210a63b8241a2233f00" member_type: VOTER last_known_addr { host: "127.0.99.132"